Nextcloud Backup System Idea



  • Since I'm planning on deploying NextCloud on my colo server, I want to also think through how I want to backup the data. In my environment there is a single 1U server.

    Thinking through RPO / RTO this is what I think is reasonable for this data. I could handle NextCloud not being available for 24 hours as well losing 24 hours worth of data. The initial amount of data that will be stored in NextCloud will be around 1.5 TB. There will likely be far less than an a gigabyte / day of new data created.

    Here is my initial idea of how to complete backups. Mind you this is a high-level picture. I haven't decided what tools would best complete this.

    1. One full backup followed by daily incremental backups to a VM on my cololab host to store the backups.
    2. Replicate the data on the Backup VM to something like Backblaze B2.

    The end result will be three copies of the data (live Nextcloud VM, Backup VM, and cloud storage), on two different media (my colocation server and the cloud storage service), with one offsite (cloud storage service).

    I think it's a good idea to do another full backup periodically (perhaps weekly or monthly). From what I understand to restore from an incremental backup, you restore the full and then all of the incrementals available from the last full backup. With taking more full backups, I have to start to consider how I can have data deduplication.

    One thing I'm considering is not having the Backup VM, and just backing up directly to some kind of cloud storage. The potential disadvantage I see in this is with recovery, since it will be faster to copy data stored on my LAN.



  • Now this is off the top of my head.
    You have restic installed on your Nextcloud VM and Backup VM.
    Restic is configured to backup the following folders:

    • config folder
    • data folder
    • theme folder

    You make a backup of your database. You can probably leave out the date +"%Y%m%d since restic timestamp your snapshots.

    mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
    

    Have restic backup your database that you've just backed up.

    • database

    After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.



  • After saying all that, now I'm interested in seeing that in action.



  • @black3dynamite said in Backup System Idea:

    Now this is off the top of my head.
    You have restic installed on your Nextcloud VM and Backup VM.
    Restic is configured to backup the following folders:

    • config folder
    • data folder
    • theme folder

    You make a backup of your database. You can probably leave out the date +"%Y%m%d since restic timestamp your snapshots.

    mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
    

    Have restic backup your database that you've just backed up.

    • database

    After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.

    I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.



  • @EddieJennings said in Backup System Idea:

    @black3dynamite said in Backup System Idea:

    Now this is off the top of my head.
    You have restic installed on your Nextcloud VM and Backup VM.
    Restic is configured to backup the following folders:

    • config folder
    • data folder
    • theme folder

    You make a backup of your database. You can probably leave out the date +"%Y%m%d since restic timestamp your snapshots.

    mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
    

    Have restic backup your database that you've just backed up.

    • database

    After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.

    I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.

    My concerns is how long you have to set Nextcloud in maintenance mode. Any new or modified data on the client will not be synced back to Nextcloud until your backup is finished. Now if you are backing up the VM itself, then it shouldn't be an issue.



  • @black3dynamite said in Backup System Idea:

    @EddieJennings said in Backup System Idea:

    @black3dynamite said in Backup System Idea:

    Now this is off the top of my head.
    You have restic installed on your Nextcloud VM and Backup VM.
    Restic is configured to backup the following folders:

    • config folder
    • data folder
    • theme folder

    You make a backup of your database. You can probably leave out the date +"%Y%m%d since restic timestamp your snapshots.

    mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
    

    Have restic backup your database that you've just backed up.

    • database

    After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.

    I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.

    My concerns is how long you have to set Nextcloud in maintenance mode. Any new or modified data on the client will not be synced back to Nextcloud until your backup is finished. Now if you are backing up the VM itself, then it shouldn't be an issue.

    For this situation, I am the only user of Nextcloud, so I could script taking it and out of maintenance mode when I'd be sleeping. I can see the concern in an environment with multiple users that could be accessing Nextcloud at any time.



  • This person create a automatic backup using restic that sends the backup to Backblaze
    It can email notifications on failure too.
    https://github.com/erikw/restic-systemd-automatic-backup



  • Here's a more straight forward auto backup to Google Drive using restic that can tweak for backing up Nextcloud VM to your Backup VM and then use the script from erikw or use the this same script to backup to Backblaze.
    https://jakobkofod.com/automated-restic-backups-google-drive/



  • @EddieJennings

    Restic does backups really well and efficiently. I use Restic on my VPS and back up to my personal OneDrive via Rclone.

    https://timothygruber.com/backup-and-restore/back-up-linux-vps-to-onedrive-via-restic-and-rclone/

    The initial backup will take a bit, but the incrementals... you won't even notice.



  • To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.



  • @DustinB3403 said in Backup System Idea:

    To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.

    NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.

    The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).



  • @EddieJennings said in Backup System Idea:

    @DustinB3403 said in Backup System Idea:

    To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.

    NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.

    The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).

    Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month



  • @travisdh1 said in Backup System Idea:

    @EddieJennings said in Backup System Idea:

    @DustinB3403 said in Backup System Idea:

    To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.

    NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.

    The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).

    Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month

    So skip having a 3rd copy of the data (live, copy on aforementioned Backup VM, copy on Backblaze storage or some other provider)?



  • As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.



  • @EddieJennings said in Nextcloud Backup System Idea:

    @travisdh1 said in Backup System Idea:

    @EddieJennings said in Backup System Idea:

    @DustinB3403 said in Backup System Idea:

    To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.

    NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.

    The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).

    Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month

    So skip having a 3rd copy of the data (live, copy on aforementioned Backup VM, copy on Backblaze storage or some other provider)?

    But you don't have a 3rd copy currently with 1 of the 3 being on the same server.

    Apparently I missed that you already have an offsite happening, so good on you.

    Being that it's your home lab, is it really worth the expense of getting a real 3rd copy? Might not matter (value of the data compared to time to restore.)



  • @travisdh1 said in Nextcloud Backup System Idea:

    But you don't have a 3rd copy currently with 1 of the 3 being on the same server.

    Yeah. Since this lab server is just a 1U in co-location, the best I could do for having an additional copy off of the Nextcloud VM is having another VM as a backup target. Of course, there's no real protection with that.

    Apparently I missed that you already have an offsite happening, so good on you.

    Nothing offsite yet. I'm trying to think all of this through before I create the Nextcloud VM and start storing data on it.

    Being that it's your home lab, is it really worth the expense of getting a real 3rd copy? Might not matter (value of the data compared to time to restore.)

    That's something I'm weighing as well. The only way I think I could get a real 3rd copy is have one on a second cloud storage provider, since having additional hardware in colo really isn't an option. I suppose I could store a copy at home as the 3rd copy, but that's more of make-a-way-to-have-a-3rd-copy-just-to-have-a-third-copy rather than as you suggested before "do it right."



  • @EddieJennings said in Nextcloud Backup System Idea:

    @travisdh1 said in Nextcloud Backup System Idea:

    But you don't have a 3rd copy currently with 1 of the 3 being on the same server.

    Yeah. Since this lab server is just a 1U in co-location, the best I could do for having an additional copy off of the Nextcloud VM is having another VM as a backup target. Of course, there's no real protection with that.

    So the question to ask is why add the complexity of backing up to a VM that would reside on the same host? It's more complex for no gain.

    Nothing offsite yet. I'm trying to think all of this through before I create the Nextcloud VM and start storing data on it.

    This is good to do.

    That's something I'm weighing as well. The only way I think I could get a real 3rd copy is have one on a second cloud storage provider, since having additional hardware in colo really isn't an option. I suppose I could store a copy at home as the 3rd copy, but that's more of make-a-way-to-have-a-3rd-copy-just-to-have-a-third-copy rather than as you suggested before "do it right."

    Or you could setup a NextCloud instance on something like Vultr and replicate between your 1U and the actual remote VM.



  • @DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.

    He absolutely should have a “backup” VM as this is what exists in reality.

    It is replicating the real world on site copy.



  • @brandon220 said in Nextcloud Backup System Idea:

    As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.

    I am surprised that they have not partnered with some provider.

    I wonder is someone could write a backup app that would backup the DB and up it to Backblaze



  • @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.

    He absolutely should have a “backup” VM as this is what exists in reality.

    It is replicating the real world on site copy.

    This was my assumption as well - he's trying to mimic real life as best he can.



  • @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.

    He absolutely should have a “backup” VM as this is what exists in reality.

    It is replicating the real world on site copy.

    Jared is correct about my intent 🙂



  • @Dashrender said in Nextcloud Backup System Idea:

    @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.

    He absolutely should have a “backup” VM as this is what exists in reality.

    It is replicating the real world on site copy.

    This was my assumption as well - he's trying to mimic real life as best he can.

    It isn’t even an assumption. It was clearly stated.



  • @JaredBusch said in Nextcloud Backup System Idea:

    @brandon220 said in Nextcloud Backup System Idea:

    As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.

    I am surprised that they have not partnered with some provider.

    I wonder is someone could write a backup app that would backup the DB and up it to Backblaze

    So mimic real fucking life and hostthe VM on a remote platform like I stated.

    See my previous post for the details as to what I said.

    Also happy new years



  • @EddieJennings said in Nextcloud Backup System Idea:

    @travisdh1 said in Backup System Idea:

    @EddieJennings said in Backup System Idea:

    @DustinB3403 said in Backup System Idea:

    To ask and to be that guy;

    Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?

    As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.

    NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.

    The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).

    Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month

    So skip having a 3rd copy of the data (live, copy on aforementioned Backup VM, copy on Backblaze storage or some other provider)?

    Number of copies is never really a factor. It's the resulting durability that matters.



  • @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch said in Nextcloud Backup System Idea:

    @brandon220 said in Nextcloud Backup System Idea:

    As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.

    I am surprised that they have not partnered with some provider.

    I wonder is someone could write a backup app that would backup the DB and up it to Backblaze

    So mimic real fucking life and hostthe VM on a remote platform like I stated.

    See my previous post for the details as to what I said.

    Also happy new years

    Again, missing the point. The Colo is replicating an on site deployment of VM server, on site primary backup and then offsite backup.



  • @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch said in Nextcloud Backup System Idea:

    @brandon220 said in Nextcloud Backup System Idea:

    As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.

    I am surprised that they have not partnered with some provider.

    I wonder is someone could write a backup app that would backup the DB and up it to Backblaze

    So mimic real fucking life and hostthe VM on a remote platform like I stated.

    See my previous post for the details as to what I said.

    Also happy new years

    Again, missing the point. The Colo is replicating an on site deployment of VM server, on site primary backup and then offsite backup.

    . . .

    download.jpg



  • @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch said in Nextcloud Backup System Idea:

    @brandon220 said in Nextcloud Backup System Idea:

    As important as backups are - I'm really surprised that NC has not integrated them into the software.
    This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.

    I am surprised that they have not partnered with some provider.

    I wonder is someone could write a backup app that would backup the DB and up it to Backblaze

    So mimic real fucking life and hostthe VM on a remote platform like I stated.

    See my previous post for the details as to what I said.

    Also happy new years

    Again, missing the point. The Colo is replicating an on site deployment of VM server, on site primary backup and then offsite backup.

    . . .

    You can be as whiny bitch as you want. it doens't change the facts of the stated case.



  • @JaredBusch The stated case is, how can I create backups and do so in a fashion that would work.

    While using the same Hypervisor and backing up to a separate VM on the same Hypervisor as the host does not create a backup.

    Since it isn't detached, I'm sure if I look hard enough I can find several instances of you spouting this.

    Even for a lab, this is a bad approach to take and the better approach would be to setup a VPS and do your backups to that as the detached backup and from that push the backups to a cloud if the goal would be to follow the 3-2-1 rule.

    I also never stated to not take backups, I implied it was stupid to take backups to the same hypervisor that your production set is made from.



  • @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch The stated case is, how can I create backups and do so in a fashion that would work.

    While using the same Hypervisor and backing up to a separate VM on the same Hypervisor as the host does not create a backup.

    Since it isn't detached, I'm sure if I look hard enough I can find several instances of you spouting this.

    Even for a lab, this is a bad approach to take and the better approach would be to setup a VPS and do your backups to that as the detached backup and from that push the backups to a cloud if the goal would be to follow the 3-2-1 rule.

    I also never stated to not take backups, I implied it was stupid to take backups to the same hypervisor that your production set is made from.

    On the contrary, a lab is exactly the place to set up something like this and test processes. That is the entire fucking point of a lab.

    Shoving off to a VPS somewhere is a totally different approach that would require different tools.

    The tools used when staying within the same network are not the same. Could they be? Sometimes.

    I don't know WTF crawled up your ass, but go take a shit and let it out.



  • @JaredBusch said in Nextcloud Backup System Idea:

    @DustinB3403 said in Nextcloud Backup System Idea:

    @JaredBusch The stated case is, how can I create backups and do so in a fashion that would work.

    While using the same Hypervisor and backing up to a separate VM on the same Hypervisor as the host does not create a backup.

    Since it isn't detached, I'm sure if I look hard enough I can find several instances of you spouting this.

    Even for a lab, this is a bad approach to take and the better approach would be to setup a VPS and do your backups to that as the detached backup and from that push the backups to a cloud if the goal would be to follow the 3-2-1 rule.

    I also never stated to not take backups, I implied it was stupid to take backups to the same hypervisor that your production set is made from.

    On the contrary, a lab is exactly the place to set up something like this and test processes. That is the entire fucking point of a lab.

    But you would never do this for a production workload. So consider the workload as any other and find the issues with the plan, punch holes in it, and realize that the plan may not hold water. A lab is a place to test process, but process needs to be considered before it is tested. One such issue with this process as discussed is if the Hypervisor or Host fails you risk everything.

    Shoving off to a VPS somewhere is a totally different approach that would require different tools.

    It's the approach that would likely be taken in cases like this where "I only have a 1U in a colo and am looking for ways to protect it".

    The tools used when staying within the same network are not the same. Could they be? Sometimes.

    I don't know WTF crawled up your ass, but go take a shit and let it out.