Promox and VM replication

scottalanmiller

@JaredBusch said in Promox and VM replication:

@scottalanmiller so if I can break you away from your Zero trust LAN-less thread.

From what I have seen you post, I know you are using PM now, and it is using LVM.
I assume in production.

Have you not done anything with replication at all? This is a pretty common use ofr virtualization platforms.

It is, but we do not. Most of our clients are small and don't want a second server. They just take backups and restore. The cost of a second server for outages that are generally less than twice a decade and typically quite minor just doesn't justify the cost. So we rarely see it.

When you need it, I think you have to stick to ZFS. Not that there is no possibility of doing it another way, but it's what ProxMox expects at this point.

scottalanmiller

There should be no issue using ZFS on top of hardware RAID so that you get blind swap AND replication.

JaredBusch

@scottalanmiller said in Promox and VM replication:

There should be no issue using ZFS on top of hardware RAID so that you get blind swap AND replication.

Yeah, one test I just set it up as ZFS, single disk.
It should have no clue what is underneath.
No matter what the fanboys say about ZFS

scottalanmiller

@JaredBusch exactly

JaredBusch

@scottalanmiller said in Promox and VM replication:

The cost of a second server for outages that are generally less than twice a decade and typically quite minor just doesn't justify the cost.

For this client, they can't take a full 24 hours of downtime before they begin to lose data. They need a replica.

It is the far remote sites that send data into the office that drop the data after 24 hours. They were never designed to hold data forever. Machines (pumps actually) with fixed capacities, etc.

There are local printouts that can be manually keyed in later in case of a real large scale disaster at the office location. But an extra $4k for a server that will soon be regionally separated makes it worth it to them.

scottalanmiller

@JaredBusch said in Promox and VM replication:

For this client, they can't take a full 24 hours of downtime before they begin to lose data. They need a replica.

Most of ours can restore from a backup in under an hour to limp along on alternative hardware. It's a pain, but they can operate a day or two that way.

DustinB3403

In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.

The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.

Dashrender

@DustinB3403 said in Promox and VM replication:

In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.

The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.

I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it

travisdh1

@Dashrender said in Promox and VM replication:

@DustinB3403 said in Promox and VM replication:

In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.

The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.

I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it

You'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.

JaredBusch

@DustinB3403 said in Promox and VM replication:

In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.

The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.

See screenshot..... It was not like hard.

JaredBusch

@Dashrender said in Promox and VM replication:

I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it

@travisdh1 said in Promox and VM replication:

You'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.

Look above.

I said it is possible I did it.
I posted a damned screenshot...

travisdh1

@JaredBusch said in Promox and VM replication:

@Dashrender said in Promox and VM replication:

I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it

@travisdh1 said in Promox and VM replication:

You'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.

Look above.

I said it is possible I did it.

I posted a damned screenshot...

I had forgotten that by the time I replied, obviously. I think it's time for me to load up Proxmox in the home lab to check out.

JaredBusch

Setup and tested.

JaredBusch

@scottalanmiller said in Promox and VM replication:

When you need it, I think you have to stick to ZFS. Not that there is no possibility of doing it another way, but it's what ProxMox expects at this point.

Sucks, but working.

JaredBusch

@JaredBusch 5 minutes later.

JaredBusch

I'll try and make a cleaner guide later..

First, spin up your ProxmoxVE system.
- During the install wizard, I left the boot drive as LVM but did change it to XFS instead of ext4. You do not setup secondary drives during the install wizard.
Once up, you need to create the ZFS store on each system, named the same, prior to joining the cluster.
- The replication process wants the ZFS pool to be named the same on both systems and you cannot name it the same (at least in the GUI) if it already exists anywhere in the cluster.
Then from the GUI, go to the disks of system 1.
then click on ZFS and create the storage pool
- name it, single disk, compression off.
wait for it to show normal.
then repeat the process on the second system.
- make sure to use the same name.
Now create the cluster and join the second system to the cluster.
You will not see the storage on ZFS storage on system 2 when it first loads up.
But it does exist if you look.
To make it available, go to storage under datacenter.
edit the existing "data" and add pve2
now it will show up and be available for use.

Is this perfect? No. but it is how I was able to get it setup to work.

The manual leave out quite a bit of specifics. on how to do things, but goes into detail on the technical of things.

Solved Promox and VM replication