Nested hypervisors?



  • Which hypervisors can be nested inside each other?
    I'm thinking primarily about KVM, Xen, Hyper-V.

    Purpose is of course for learning and playing with hypervisors, not production.





  • @pete-s

    VMware Workstation and Player can do it, so you dont have to bother with getting extra separated hardware.



  • @pete-s said in Nested hypervisors?:

    Which hypervisors can be nested inside each other?
    I'm thinking primarily about KVM, Xen, Hyper-V.

    Purpose is of course for learning and playing with hypervisors, not production.

    I know for a fact Hyper-V, KVM, and VMWare do it.



  • @emad-r said in Nested hypervisors?:

    @pete-s

    VMware Workstation and Player can do it, so you dont have to bother with getting extra separated hardware.

    Does the free version of Workstation Player allow nested hypervisors?



  • Xen provides nesting.
    Hyper-V provides nesting.
    KVM provides "working but experimental" nesting.



  • @black3dynamite said in Nested hypervisors?:

    Workstation Player

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011
    https://www.virtualbox.org/ticket/4032

    0_1534886780103_2018-08-22 00_26_03-Windows 10 - VMware Workstation.png



  • @scottalanmiller said in Nested hypervisors?:

    KVM provides "working but experimental" nesting.

    Which is fine... because I can't imagine having to use it outside of testing or lab scenarios.

    I didn't know KVM still had it in experimental mode:

    https://www.linux-kvm.org/page/Nested_Guests

    For completeness:
    https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/


  • Vendor

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Virutalbox is kinda useless as basic things like USB 3 drivers require a 5000$ license.

    ESXi has supported nesting longer than all of them, and has a few production use cases for hypervisors on hypervisors.



  • @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?



  • @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .



  • @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?



  • @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    Vendors never require a specific hypervisor. If they did, you certainly wouldn't nest it. And if it was a consideration, you'd find a different vendor.

    Really, it's for lab/testing.



  • @obsolesce said in Nested hypervisors?:

    Vendors never require a specific hypervisor.

    That's actually not true. I have seen at least one that required ESXi, but you could make the caveat that there are no GOOD vendors.



  • @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.



  • @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    I haven't ever had the need to even look at attempting this so I don't know. But lets say you're a Hyper-V shop and have a business requirement you have to run an appliance of some kind that is tailored to ESXi, this would be a case where you'd likely nest.

    Rather than building another hypervisor fleet.



  • @obsolesce said in Nested hypervisors?:

    Really, it's for lab/testing.

    Alright, I meant real-world but I didn't say that. That's on me.



  • @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Is it common for a Dev to need access to the hypervisor themselves? Sounds weird but I don't work with devs a lot (yet)



  • @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?



  • @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    I haven't ever had the need to even look at attempting this so I don't know. But lets say you're a Hyper-V shop and have a business requirement you have to run an appliance of some kind that is tailored to ESXi, this would be a case where you'd likely nest.

    Rather than building another hypervisor fleet.

    Nested (virtual) ESXi is not officially supported by VMWare, so that's not a production scenario anyways.



  • @dustinb3403 said in Nested hypervisors?:

    @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

    I mean, isn't it likely to be his own host completely? I wouldn't let someone manage a host I'm responsible for. At that point he just manages it right? No need for nesting.



  • @dustinb3403 said in Nested hypervisors?:

    would nesting be required?

    Only if full hypervisor control is required. I almost did this for a dev... well I did, but it turned out in the end that he wanted Virtualbox... so that's what he got.



  • @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

    I mean, isn't it likely to be his own host completely? I wouldn't let someone manage a host I'm responsible for

    Well. . . no

    Just as an example, with XenServer (and XO) you can create users and give them access to a specific pool or set amount of resources on any server in the pool, and to what guests they could affect.

    So this would allow the user to do their job without the need for additional hardware or nesting. Unless their job was to develop on a specific hypervisor.



  • @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

    I mean, isn't it likely to be his own host completely? I wouldn't let someone manage a host I'm responsible for

    Well. . . no

    Just as an example, with XenServer (and XO) you can create users and give them access to a specific pool or set amount of resources on any server in the pool, and to what guests they could affect.

    So this would allow the user to do their job without the need for additional hardware or nesting. Unless their job was to develop on a specific hypervisor.

    Gotcha. Haven't ever needed to use that functionality. I was worrying about stuff like thin-provisioning but if you can limit the resources that's great.



  • Which I would be wary of anyone who says "I must do my development work on <insert hypervisor>". Because I know they are almost certainly doing the work within a guest and are just comfortable with the tools.



  • @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

    I mean, isn't it likely to be his own host completely? I wouldn't let someone manage a host I'm responsible for

    Well. . . no

    Just as an example, with XenServer (and XO) you can create users and give them access to a specific pool or set amount of resources on any server in the pool, and to what guests they could affect.

    So this would allow the user to do their job without the need for additional hardware or nesting. Unless their job was to develop on a specific hypervisor.

    To get that kind of control wiht Hyper-V , you need SCVMM.

    We don't have that anymore.

    On top of that, he was good with KVM. So I had created a nested KVM host on Hyper-V which was great for him for a while. But he was coming from VirtualBox, and didn't have time to convert all the stuff and whatever else that was involved.



  • @obsolesce yeah I'm not saying there isn't a reason or whatnot for using nested hypervisors. Just discussing the possible use cases I could imagine for it and where it may or may not make sense.



  • @obsolesce said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @obsolesce said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @dustinb3403 said in Nested hypervisors?:

    @wirestyle22 said in Nested hypervisors?:

    @emad-r said in Nested hypervisors?:

    Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    If a vendor is stating that they only support a specific guest OS on a specific Hypervisor they had better be supporting the entire stack and not just an application that is on the guest. . .

    What scenario is a nested hypervisor useful in any way?

    The only realistic "production" usage for nesting, would be if you for example want to give a Dev his/her own hypervisor to cycle through VMs... or some similar situation where you can't dedicate hardware to.

    Even in this case, would nesting be required?

    Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

    I mean, isn't it likely to be his own host completely? I wouldn't let someone manage a host I'm responsible for

    Well. . . no

    Just as an example, with XenServer (and XO) you can create users and give them access to a specific pool or set amount of resources on any server in the pool, and to what guests they could affect.

    So this would allow the user to do their job without the need for additional hardware or nesting. Unless their job was to develop on a specific hypervisor.

    To get that kind of control wiht Hyper-V , you need SCVMM.

    We don't have that anymore.

    On top of that, he was good with KVM. So I had created a nested KVM host on Hyper-V which was great for him for a while. But he was coming from VirtualBox, and didn't have time to convert all the stuff and whatever else that was involved.

    AzMan (Authorization Manager) was the method we used to delimit host access permissions up until 2012 R2. It's been deprecated which is sad as it actually worked quite well.

    On 2016 going forward, SCVMM and I think Windows Admin Center may also have tiers but we've not really had any time to invest in WAC as of yet.



  • @wirestyle22 said in Nested hypervisors?:

    Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

    learn and test

    https://mangolassi.it/topic/17807/proud-smug-post-kvm-and-gluster-and-accomplishment/1



  • @wirestyle22 said in Nested hypervisors?:

    Maybe

    Guest Clusters have been around for a while on the Hyper-V platform. Backing up the shared storage being used by the guest clusters is another matter that has not been solved as of yet. At least, not cleanly.

    There are some business cases for guest clusters such as Exchange or SQL whose teams require their own resilience measures be in place in order for the setup to be "supported" in the event something goes wrong.

    EDIT: Meh … low blood sugar … need to eat as it's been a busy day already. I totally missed "guest cluster" versus "nested hypervisor". 😛