No way to create larger than 2TB virtual disk with Xen or XCP-NG?
-
Well I went ahead and used the "Experimental" EXT4/XFS drivers. They work fine from what I can tell. I actually used XFS and not EXT4. I haven't installed an OS on EXT4 in years.
But you are still limited to 2TB for the VHD size no matter the underlying file system. So I had to create multiple 2TB vhds with xcp-ng for that SR, attach those vhds to the VM, and then span the volumes in the VM guest to create one large fat drive. It's not quite an elegant approach but I guess it works nonetheless.
One thing I don't like is xcp-ng doesn't make it very easy to figure out how to use all the available space on the SR when creating the vhds. For example, after I added my 6TB SR, I needed to break that up into 3 vhds. So I created two 2TB vhds and needed one more to use the remaining space. However, you have to do some math here to figure out how much space you have left as I noticed xcp-ng will let you overprovision the SR to your heart's content. I could have created a dozen 2TB vhds on the single 6TB storage repo if I wanted to. Also, when creating vhds, it asks you how large you want the vhd to be in GB (Gigabyte) but after you create it, its shows you the size in TiB (Tebibytes).
-
@Pete-S said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Is there no way to just pass the whole disk right to a VM?
Yes, there is. You need to edit a config file on the host. Then the disk will show up as removable storage in xenserver/xcp-ng and you can assign it to the VM you want.
BTW, it's the block device that you will pass from the hypervisor to the guest. So it can be any size and any file system that the guest supports.
It also works with drives that are already formatted, for instance if you insert an NTFS drive full of files to your xenserver host you can pass it to a windows guest VM.
-
@Pete-S
Pete, I actually found your post (at least I think it was from you, I was tired last night) from a year ago or so on here about doing exactly that. I may test that method out as well. -
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Well I went ahead and used the "Experimental" EXT4/XFS drivers. They work fine from what I can tell. I actually used XFS and not EXT4. I haven't installed an OS on EXT4 in years.
Hey awesome, that's way better.
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
But you are still limited to 2TB for the VHD size no matter the underlying file syste
Lame
-
@black3dynamite said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
2TB limit is because of ext3.
Wow, I haven't used ext3 in ages lol.
-
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@black3dynamite said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
2TB limit is because of ext3.
Wow, I haven't used ext3 in ages lol.
It's still what XO uses. (shakes head)
-
@scottalanmiller said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@black3dynamite said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
2TB limit is because of ext3.
Wow, I haven't used ext3 in ages lol.
It's still what XO uses. (shakes head)
I think you meant XS/XCP-ng, correct?
-
Yeah its strange that Xen's base file system is still ext3. I mean, that is very very old. Wonder what the holdup is to move it to at the very least ext4?
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Yeah its strange that Xen's base file system is still ext3. I mean, that is very very old. Wonder what the holdup is to move it to at the very least ext4?
yeah, for sure.
-
@Danp said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@black3dynamite said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
2TB limit is because of ext3.
Wow, I haven't used ext3 in ages lol.
It's still what XO uses. (shakes head)
I think you meant XS/XCP-ng, correct?
Sorry, XCP-NG I meant. Made by XO
-
@scottalanmiller said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Sorry, XCP-NG I meant. Made by Vates, the creator of XO
FTFY!
-
@scottalanmiller said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@Obsolesce said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@black3dynamite said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
2TB limit is because of ext3.
Wow, I haven't used ext3 in ages lol.
It's still what XO uses. (shakes head)
It's still what XenServer uses. XO is just management.
I doubt the XO people have enough resources to reprogram that entire stack yet. We know everyone wants just that.
-
Thanks for correcting the sentence @travisdh1
Indeed, SMAPIv1 is using VHD format everywhere. This format is limited at 2TiB by "design" [1] . This has nothing to do with XO or even XCP-ng because it's a fork of XenServer, ie a copy with new or improved code. So remember that regardless which filesystem you use, as long as you are using VHD format to store virtual disk, you are limited to 2TiB.However, SMAPIv3 is using
qcow2format instead, "solving" this limitation. We (XCP-ng team) are currently working on improving SMAPIv3 to support disk import/export inqcow2(which isn't even done by Citrix people themselves
). As soon we got that, the next step is to write drivers for ext4for example, which is doable relatively easily.One of main issue with SMAPIv3 (there's others) is the fact a part of the development is done privately by Citrix instead of collaborating (see this conversation on GitHub), so the goal is to catching up on our side to be able to get an upstream public faster and become the de facto upstream standard. We are working toward that but it's not something you solve in one week (you need to go deep in qemu-dp/xen blktap, see our efforts here etc.)
[1]: The VHD format has a built-in limitation of just under 2 TiB (2040 GiB) for the size of any dynamic or differencing VHDs. This is due to a sector offset table that only allows for the maximum of a 32-bit quantity. It is calculated by multiplying 232 by 512 bytes for each sector.
edit: also, as soon we got
qcow2import/export support in XCP-ng, we could use that format in XO to store backup. So far, there's only 2 options to get disk data from XS/XCP-ng: raw or vhd (that's why XO is storing VHD files, because… that's what we got from the hypervisor!) -
@olivier Wow thanks for that very detailed response. It’s a shame that Citrix isn’t playing nice with SMAPIv3. But it’s also great to hear you guys are working on it anyway!
I’m really impressed with the entire xcp-ng project. Really amazing some of the changes you guys have brought. You seem to have an excellent team of devs.
-
@olivier I was reading over the Citrix docs and it looks like you can only attach a maximum of 7 VHDs to a single VM. Simple math would tell us that the maximum VM size one could ever build with XEN (xcp-ng) could only be 14TB in size (7 VHD * 2TB each) since we are also limited to a 2TB max VHD size.
Does this sound right? How are people creating larger VMs??? What am I missing?
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
Does this sound right? How are people creating larger VMs??? What am I missing?
Very few do, that's an enormous VM. For the rare need, they attach directly I'd assume.
-
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
-
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
-
@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:
@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.
Why stay with XCP-NG? why not move over to KVM?
It's a good point. Seems like a generic KVM install would meet your needs out of the box, super simply.
