No way to create larger than 2TB virtual disk with Xen or XCP-NG?
-
Thanks for correcting the sentence @travisdh1
Indeed, SMAPIv1 is using VHD format everywhere. This format is limited at 2TiB by "design" [1] . This has nothing to do with XO or even XCP-ng because it's a fork of XenServer, ie a copy with new or improved code. So remember that regardless which filesystem you use, as long as you are using VHD format to store virtual disk, you are limited to 2TiB.However, SMAPIv3 is using
qcow2format instead, "solving" this limitation. We (XCP-ng team) are currently working on improving SMAPIv3 to support disk import/export inqcow2(which isn't even done by Citrix people themselves
). As soon we got that, the next step is to write drivers for ext4for example, which is doable relatively easily.One of main issue with SMAPIv3 (there's others) is the fact a part of the development is done privately by Citrix instead of collaborating (see this conversation on GitHub), so the goal is to catching up on our side to be able to get an upstream public faster and become the de facto upstream standard. We are working toward that but it's not something you solve in one week (you need to go deep in qemu-dp/xen blktap, see our efforts here etc.)
[1]: The VHD format has a built-in limitation of just under 2 TiB (2040 GiB) for the size of any dynamic or differencing VHDs. This is due to a sector offset table that only allows for the maximum of a 32-bit quantity. It is calculated by multiplying 232 by 512 bytes for each sector.
edit: also, as soon we got
qcow2import/export support in XCP-ng, we could use that format in XO to store backup. So far, there's only 2 options to get disk data from XS/XCP-ng: raw or vhd (that's why XO is storing VHD files, because… that's what we got from the hypervisor!) -
@olivier Wow thanks for that very detailed response. It’s a shame that Citrix isn’t playing nice with SMAPIv3. But it’s also great to hear you guys are working on it anyway!
I’m really impressed with the entire xcp-ng project. Really amazing some of the changes you guys have brought. You seem to have an excellent team of devs.
-
@olivier I was reading over the Citrix docs and it looks like you can only attach a maximum of 7 VHDs to a single VM. Simple math would tell us that the maximum VM size one could ever build with XEN (xcp-ng) could only be 14TB in size (7 VHD * 2TB each) since we are also limited to a 2TB max VHD size.
Does this sound right? How are people creating larger VMs??? What am I missing?
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Does this sound right? How are people creating larger VMs??? What am I missing?
Very few do, that's an enormous VM. For the rare need, they attach directly I'd assume.
-
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
-
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
It's a good point. Seems like a generic KVM install would meet your needs out of the box, super simply.
-
You can attach more than 7 disks when you have tools in the VM. In your case, you don't need a VM in the traditional way, ie something flexible that you can migrate etc. So you can indeed attach your disks directly, regardless the hypervisor you choose.
Another more flexible alternative would be to have a "normal" VM, but attach a NFS share on it to store your data. This way you keep the flexibility of the VM and the large storage you need. The extra requirement is any NFS capable machine (even a very cheap NAS)
-
You don't HAVE to virtualize despite the best practice. In some scenarios the major benefits simply don't apply and running on hardware can be an easier server management choice with more direct benefits and/or easier backup/restore options.
I'm not completely aware of your whole scenario and environment, so I'm just putting this out there... not as a suggestion.
-
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
I've used KVM and really like it. Been running it on a Debian host for yeras. For the new host build, I wanted to try something different mainly. I was impressed with Xen and when I found out about xcp-ng I had to give it a spin.
@olivier Great! I think I will just pass through the disks and be done with it. I could have gone the NAS route, but I have enough local storage to not have to use that option.
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
I've used KVM and really like it. Been running it on a Debian host for yeras. For the new host build, I wanted to try something different mainly. I was impressed with Xen and when I found out about xcp-ng I had to give it a spin.
@olivier Great! I think I will just pass through the disks and be done with it. I could have gone the NAS route, but I have enough local storage to not have to use that option.
Frankly, @Obsolesce might be the rightest here. Perhaps don't virtualize. Your storage requirement makes migrating to another platform more challenging - not saying not doable. and you plan to have a dedicated machine - with, now, likely directly provisioned storage due to the hypervisor of choice limitations.
It's one thing to want to get more experience at something - but at the cost of an enterprise implementation seems foolish at best. Making the the most sound IT decision for the job at hand should be the goal - not furthering your education (that's what labs are really for).
-
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
I've used KVM and really like it. Been running it on a Debian host for yeras. For the new host build, I wanted to try something different mainly. I was impressed with Xen and when I found out about xcp-ng I had to give it a spin.
@olivier Great! I think I will just pass through the disks and be done with it. I could have gone the NAS route, but I have enough local storage to not have to use that option.
Frankly, @Obsolesce might be the rightest here. Perhaps don't virtualize. Your storage requirement makes migrating to another platform more challenging - not saying not doable. and you plan to have a dedicated machine - with, now, likely directly provisioned storage due to the hypervisor of choice limitations.
It's one thing to want to get more experience at something - but at the cost of an enterprise implementation seems foolish at best. Making the the most sound IT decision for the job at hand should be the goal - not furthering your education (that's what labs are really for).
I have no desire to migrate this VM. It's just running a camera recording software suite for my business. Honestly, it's nice to have but not a necessity. I used to run it virtualized via KVM as well and also passed through the disks to make managing that much storage easier.
At any rate, disk pass through was successful for xcp-ng. I appreciate everyone's input.
-
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
In some scenarios the major benefits simply don't apply and running on hardware can be an easier server management choice with more direct benefits and/or easier backup/restore options.
The top benefits always apply. Its a myth that they do not. Abstraction and stability don't change with the workload. There are no easier backup/restore options, only fewer.
-
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Your storage requirement makes migrating to another platform more challenging - not saying not doable. and you plan to have a dedicated machine - with, now, likely directly provisioned storage due to the hypervisor of choice limitations.
None of those things would be factors in deciding to not virtualize, though. All red herrings. The reason to not go to physical is the lack of benefits. Never try to show why a best practice "is better enough", you have to actually show how the alternative (physical) in this case would be superior. But no one has mentioned a single reason why it is better, just that it "isn't that bad." This is extremely bad logic.
-
@scottalanmiller The only real "benefit" of physical that I can tell for my specific use case would be that I can take advantage of Quick Sync since there doesn't appear to be a way to pass that through the Xen hypervisor (at least not that I have found without putting on my hacking hat).
But I really don't want to have to have a physical host just for that use. I still want to virtualize a few other things I host internally so I'd have to now have two hosts. One for the cams and one as a VM host. It's just more complication than I want to have to deal with to be honest.
@olivier Made an interesting suggestion in that I could setup a NFS share on another host. But then I'm back to two hosts again (or a Synology NAS).
I'm just trying to keep things a bit simple. I don't think I can get much simpler that using one host and simply passing through a couple gigantic HDDs for one specific VM. I think the VM could still be migrated so long as I detach the passthrough disks first, move those disks to a new host, migrate the VM to the new host, and then re-attach/passthrough the disks on the new host. At least, I don't see why I couldn't do this if I ever wanted to move to another host in the future. Going purely physical would make that more of a challenge I think.
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@olivier Made an interesting suggestion in that I could setup a NFS share on another host. But then I'm back to two hosts again (or a Synology NAS).
I wonder - can you create an NFS mount point in XenServer or XCP-NG? then just share that via loopback?
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
I'm just trying to keep things a bit simple. I don't think I can get much simpler that using one host and simply passing through a couple gigantic HDDs for one specific VM. I think the VM could still be migrated so long as I detach the passthrough disks first, move those disks to a new host, migrate the VM to the new host, and then re-attach/passthrough the disks on the new host. At least, I don't see why I couldn't do this if I ever wanted to move to another host in the future. Going purely physical would make that more of a challenge I think.
I'm curious - would these large disk be part of a RAID array, or JBOD?
-
@Dashrender JBOD for the raw camera storage. I don't need/want any redundancy here.
-
@scottalanmiller said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
In some scenarios the major benefits simply don't apply and running on hardware can be an easier server management choice with more direct benefits and/or easier backup/restore options.
The top benefits always apply. Its a myth that they do not. Abstraction and stability don't change with the workload. There are no easier backup/restore options, only fewer.
Generally... yes.
It all depends on how it's done from start to finish. I could potentially build a nonvirtualized setup using a DAS with a great hybrid cloud continuity strategy that is more stable, reliable, and performant than someone else's shitty virtualized setup with pass-through disks and bad backup/DR strategy. The reverse applies as well.Sometimes hardware features or requirements take virtualization out of the equation, or don't make sense.
Virtualization itself I don't think automatically guarantees more stability than a non virtualized system. I can see how it can (or should) make a system more reliable though.
-
I think the VM could still be migrated so long as I detach the passthrough disks first, move those disks to a new host, migrate the VM to the new host, and then re-attach/passthrough the disks on the new host.
You can indeed. Not very practical but no technical barrier.
I wonder - can you create an NFS mount point in XenServer or XCP-NG? then just share that via loopback?
I don't really see the point of doing that? I had in mind an NFS share mounted directly in the VM. Simple, efficient (if you already have a NAS obviously)
