@scottalanmiller said in Nested hypervisors?:

This would work, as only one vendor supports the virtualization of another hypervisor (and only for a niche use case) that I'm aware of.

Most vendors specify that X hypervisor must be there, and specify nothing of what is beneath it.

Beyond the performance hit on the IO path (lot of interrupts make this not efficient) this is the same as running on non-supported hardware.

If a vendor uses Oracle DB and says "We use Oracle" and you run Oracle on a x86 emulator on a Raspberry Pi, I can see you trying to claim a loophole, and I can also see the vendor not wanting to support you.

@wirestyle22 said in Nested hypervisors?:

I've never seen a application vendor that mandates a hypervisor, but doesn't care if it's deployed in a non-supported configuration. I could see EPIC and the like having a good laugh though if you ask them.

You mean you've never seen a good vendor do that right? I've definitely seen it.

Who is a better EMR vendor than EPIC? They are kinda the gold standard and the #2 (Cerner) isn't going to support it either.
Even if we get into the smaller players (Care4, AllScripts) that are not supported either. That's also ignoring that the DB vendors in these cases (Cache, Oracle, etc) are going to not support it.

Try telling a chief medical officer, or head a practice "Hey... so we are going to go with this no-name vendor for the application you spend 90% of your time in, because they would support Hyper-V on KVM on ESXi!"

What vendor and what nesting have you seen supported?

@scottalanmiller said in Nested hypervisors?:

@storageninja said in Nested hypervisors?:

@dustinb3403 said in Nested hypervisors?:

I haven't ever had the need to even look at attempting this so I don't know. But lets say you're a Hyper-V shop and have a business requirement you have to run an appliance of some kind that is tailored to ESXi, this would be a case where you'd likely nest.
Rather than building another hypervisor fleet.

This wouldn't be supported by any VMware appliance vendor I"m aware of, and VMware certainly wouldn't support it.

But would be supported by the application vendor.

I've never seen a application vendor that mandates a hypervisor, but doesn't care if it's deployed in a non-supported configuration. I could see EPIC and the like having a good laugh though if you ask them.

@scottalanmiller said in What Is an Agentless Backup:

But that could change if a vendor cared to partner with them.

The only implementations I've seen on this for KVM, involved a fork of it with proprietary API's. Does Scale have a standardized CBT/VADP like API system in place to support 3rd party backup hooks?
I asked some backup vendors about KVM support a while back and they all pointed to the lack of a common standard for robust, stable, consistent API's for this.

@scottalanmiller said in What Is an Agentless Backup:

Comparing the Scale "snapshot" vs Veeam "snapshot" is an apples-orange comparison.

Correct, as Veeam doesn't have snapshots. It relies on the snapshots coming from the hypervisor. So Veeam has to use whatever snapshot mechanism is already there, whereas Scale provides the snapshot themselves.

Veeam can also manage snapshots at the storage array level (either using vVols that pass them to the array or through 1/2 a dozen storage vendors they have API integrations with).

@dustinb3403 said in What Is an Agentless Backup:

Maybe an agent based solution would be substantially better, because it's "all gui's" at that point and the "Windows Guys" just need to point and click to restore a file, folder or an entire system.
Taking each part of this conversation as "the only thing that matters now" and discussing it individually without considering the entire picture is a foolish task. As almost certainly there will be a follow up conversation of "why doesn't it do this".

Quite true. One thing to note is some agentless systems can offer the "windows guy" hat point and click File Level Recovery (FLR) or application level recovery (RMAN integration for say Oracle) to that DBA who has his own toolchain he likes to use.

@dustinb3403 said in What Is an Agentless Backup:

If your RPO and RTO objects are lax (say 48 hours of downtime) any storage vendor in the world should be able to at a minimum, ship you a copy of your data.

Got 100TB that you trickled in using differentials or a seed on a 1.5Mbps T1? Don't underestimate the Glacier of cold data beneath the water that will take forever to pull back.

Now this doesn't mean that you'll have somewhere to restore it too, or run it from. But that of course is a part of your recovery and restoration plan. Having a facility that is capable of supporting your staff and systems, a supplier (or spare hardware in working condition), accessibility to that space and systems.

Spot on. The planning (and TESTING of that plan) are the big thing. I've seen many people who had SAN snapshots who discovered it took them 3 days to make them useable when something went boom.

@momurda said in What Is an Agentless Backup:

@storageninja said in What Is an Agentless Backup:

@momurda said in What Is an Agentless Backup:

I dont get the controversy here. Every single agentless backup uses snapshots, always has. Veeam, Unitrends, etc. How else could they work?
edit: For example, I can do a snapshot in XS/xcp, then export that snapshot. That is a real agentless backup. This is exactly the same process Unitrends uses, with a bit of flair added on like dedup and some other stuff like automation.

This is actually incorrect. There have been agentless systems that can mirror data without using a snapshot by leveraging write splitting technology. RecoveryPoint was an early one in the physical layer (and similar storage virtualization engines). VAIO based replication (RP4VM's, Veritas) also can replicate without a snapshot as the API's allow for write splitting to occur at the hypervisor layer giving you access to a "journal" and window you can recover from.

You may use snapshots, or scripts to stun applications WITH these technologies to improve consistency of recovery, but they have existed for a long time and can run without snapshots.

Uhhuh. What is the cost of these solutions? Hundreds of thousands of dollars? Millions of dollars? This is an SMB IT admin forum, not Fortune 100 IT Admin forum.

RecoverPoint4VM's is licensed per VM. While it's not cheap, it's not hundreds of thousands of dollars. Just because a company is small doesn't mean it might have a need WAN-efficient low RPO (What their specialty is). I worked for a 50 man call center, and I had systems that if they were down for 15 minutes it could lead to catastrophic repercussions (we did dispatching for medical messaging, and organ transport services).

Any of the solutions you mention here are meant for huge shops with annual multimillion dollar IT budgets.

I know a small credit union using them. Is someone with 30VM's HUGE with multi-million IT budgets?

I could give you specifics, but like all IT products, the price is hidden until you talk to someone on the phone.

And like all IT products, a quick chat with a VAR will give you budgetary pricing.

I doubt anybody here on this forum is in an environment where these would be considered. One of the examples I see on the web for one of these products is for 2500 vms.

Just because a solution will scale to 2500 VM's doesn't mean it can't scale down.

None of them work with anything other than VMware.

Veeam works with Hyper-V and Azure, as well as Amazon EC2 via their last acquisition. Dell-EMC RecoverPoint actually predates x86 virtualization being popular and works just fine for Windows, Linux etc.

The nature of IT is systems and solutions that only worked for multi-million dollar projects eventually trickles down to the humble peasants. When I worked in SMB IT in 2007 when Enterprise hypervisors that were production ready became "Free" I jumped to deploy them. The tooling you use today was once something that a company of your scale likely couldn't afford.

I had a chat today with a grocery store chain, who considered IT a "cost" and "We sell tortillas and beer" was I'm sure a response to IT spend. They recently acquired a startup and are now struggling to keep up with containers and devops requests. All it takes is growth in the right area, or the right M&A and overnight what you need to be familiar with could change radically. Everyone company is an IT company these days it seems...

@dustinb3403 said in Nested hypervisors?:

I haven't ever had the need to even look at attempting this so I don't know. But lets say you're a Hyper-V shop and have a business requirement you have to run an appliance of some kind that is tailored to ESXi, this would be a case where you'd likely nest.
Rather than building another hypervisor fleet.

This wouldn't be supported by any VMware appliance vendor I"m aware of, and VMware certainly wouldn't support it.

@obsolesce said in Nested hypervisors?:

Nested (virtual) ESXi is not officially supported by VMWare, so that's not a production scenario anyways.

Technically it's used for the vSAN Witness (which is just a ESXi VM). It's used internally for QE testing, It's used for the hands on labs also.

@scottalanmiller said in Nested hypervisors?:

Why not do permission based limitations so you can provide a Dev with access to create/destroy as many VM's as he/she needs within the constraints of your pool or resource limits?

This is commonly done with a Cloud Management Product that overlays the hypervisor (OpenStack, vRealize Autiomation, etc, PKS etc) that can manage the multi-tenancy etc.
Now some hypervisors support some provisioning options others don't (Instant Clones are a unique one, that allow cloning a running VM with thin memory) that may make some things more efficient

@obsolesce said in Nested hypervisors?:

@wirestyle22 said in Nested hypervisors?:

@emad-r said in Nested hypervisors?:

Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

Maybe I'm missing something but why in the world would I ever want to use nested hypervisors? Vendor requirements?

Vendors never require a specific hypervisor. If they did, you certainly wouldn't nest it. And if it was a consideration, you'd find a different vendor.

Really, it's for lab/testing.

This would work, as only one vendor supports the virtualization of another hypervisor (and only for a niche use case) that I'm aware of.

@fateknollogee said in What Is an Agentless Backup:

Do you need another Scale device on the other end or not?
With Veeam (in a non-Scale enviroment), I can send the "snaphots/backups" (or whatever you want to call them) to a generic server/NFS/SMB share etc.

This is the core of the TCO battle between "Storage Appliances" and Storage/Backup software. One requires you buy xxx appliance where the software is tied to the hardware. this is fine if forward pricing can be guaranteed at a consistent level, or the ease of "Single SKU purchase/sale" exceeds the lack of cost control. You couldn't replicate a VMAX to a DL380, and nothing is really new here.

Another thing is if you replicate it to "something else" I also need to consider what I need to recover too. Veeam with their DRaaS partners will let you recover from HPE to Dell, or Cisco to Lenovo or whatever the partner is using for hosting. I think they have over a 1000 hosting partners. They even offer the ability to restore to various public clouds (IBM Cloud, VMC on AWS, Azure). IF you are operating a hybrid cloud strategy with one of these providers already (and have the direct connections etc) that can help with the decision. WHERE you are going to put the data is only as important as WHAT you are going to do with it.

Just because a storage/backup solution can send data to a cloud (the cheap part) doesn't mean it can recover local to that cloud, and if you are stuck waiting on the data to egress back out to a copy of that appliance it can cause issues with hitting your RTO.

@momurda said in What Is an Agentless Backup:

I dont get the controversy here. Every single agentless backup uses snapshots, always has. Veeam, Unitrends, etc. How else could they work?
edit: For example, I can do a snapshot in XS/xcp, then export that snapshot. That is a real agentless backup. This is exactly the same process Unitrends uses, with a bit of flair added on like dedup and some other stuff like automation.

This is actually incorrect. There have been agentless systems that can mirror data without using a snapshot by leveraging write splitting technology. RecoveryPoint was an early one in the physical layer (and similar storage virtualization engines). VAIO based replication (RP4VM's, Veritas) also can replicate without a snapshot as the API's allow for write splitting to occur at the hypervisor layer giving you access to a "journal" and window you can recover from.

You may use snapshots, or scripts to stun applications WITH these technologies to improve consistency of recovery, but they have existed for a long time and can run without snapshots.

@scottalanmiller said in What Is an Agentless Backup:

@fateknollogee said in What Is an Agentless Backup:

I don't even think the Scale guys could/would make that claim.

They don't in their whitepaper, but might as well, as they state that for special cases and workloads that agent-based would be an alternative to the built-in backup system. As agentless is the only alternative to agent-based, and built in platform backups are called agentless, they've made it clear that that is what they are considering it, which they should, as that is clearly what it is.

https://www.scalecomputing.com/documents/white-papers/white_papers_backup_hc3.pdf

They can take a snapshot and replicate it somewhere else. That isn't really revolutionary in and of itself (EMC had storage arrays that could do this in the 90's). One challenge to this is what are you recovering?

1. does the application require being put in a backup ready state? (Does the snapshot system support running pre-freeze and post, thaw scripts in the guest to accomplish this?). more importantly is this something I"m going to manually have to do for each application, or does the backup software detect and have awareness of how to do this and report on success/failure?

2. Agentless backups that just block level clones may be useful for some things (Image level restore of an OS drive) but there are other things people often look for. Support for file level, or application level (Brick level sometimes called) restores? Index's built to make the search for that possible? the ability to "search" all the snaps without having to mount them (Veeam Array offload snap protection with Nimble as an example) can mean the difference between 5 minutes, and hours of fighting with an application to get an export of a schema to get some things restored.

3. Testing. Can the Backup and DR platform perform fully isolated testing of recovery (Build an isolated shadow network that's segmented, perform application and service checks). This is where the DR/BC software gets pretty nifty and advanced on making runbook testing easy.

4. Does the application have distributed state? If so you may need to snapshot and quiescent and stun a GROUP of VM's at the same time. There's some real devil in the details with this stuff, but new Scale-out apps this is also a concern.

@brandon220 said in How to take advantage of virtualization. Major products get updated:

I plan on setting this up in the lab soon. dm-cache also sounds interesting. I've never touched software RAID because 95% of my environment has been MS for a long time. Always have gone the hw raid route.

https://www.redhat.com/en/blog/improving-read-performance-dm-cache

Redhat did some testing with an SSD but it looks ugly. 5 passes and no performance improvement using a SSD. I suspect they are hamstrung by the patent minefield that is ARC (IBM of all people has this patent BTW) and the subsidiary cool optimizations that have been made to it (ARC was intended for CPU cache originally, your storage fun fact of the day!). Also I suspect the IO path on this thing isn't the cleanest. Looks like the Linux kernel file cache is going to be faster which if I"m using that I might as well just give memory to the guest and let it sort it out (especially with the lack of dedupe or single instancing of this cache).

If your looking to speed stuff up I say get "the good stuff".

We got some PMEM DIMMS in the lab, and this stuff is face melting fast. You can "bolt" it on with a DAX file system, but the best way to use it is with applications that have been redesigned to support it. We forked REDIS to support this and got latency 12x better than using local NVMe drives, and 2.8x better tha DAX.
Oracle had 57x better operational latency.

@scottalanmiller said in How to take advantage of virtualization. Major products get updated:

I just did a new install this week and it turned out actually easier than even VMware at this point! Which is amazing because Vmware is so easy.

Attach ISO and mash enter and F2 once was what I joking called the ESXi installer back in the day.
Dell/HPE/Cisco/Lenovo/SuperMIcro will PRE-INSTALL ESXi to a M.2 mirror or SD cards. "It was already installed on the damn box" is hard to beat.

If your talking about a AutoDeploy ESXi supports PXE booted stateless hosts also.
And that's before we look at templated deployments using Razor/Puppet/OpenManage/UCS that exist.

Now you could do the same with other OS's (and use the state tool of your choice). Hell I've seen people use DSC to manage ESXi hosts even (weird, but it's a thing I guess).

@travisdh1 said in How to take advantage of virtualization. Major products get updated:

Vmware ESXi provides VVOLS and VAAI technologies that no other virtualization products provide.

Nobody else calls it the same thing, but the same functionality is available is always available to use.

vVols (especially the block implementation that allows sub-LUN object creation against block) isn't replicated anywhere else. VASA-3 isn't replicated anywhere else. Some of the VAAI primates are represented in other places (XCOPY being a common one) but some of the others don't exist elsewhere (the NFS primates, TP_STUN) or lack quite a bit of finesse in implementation (UNMAP). This also gets back to "someone checked in a module" vs. a 3rd party with enterprise support will actually support you enabling that feature (Xen Remus was a great example of this).

VASA as a 2 way control channel is pretty badass. Cinder/SMI-S are NOT the same thing. There isn't a end to end control, manage, monitor framework like it.

@travisdh1 said in How to take advantage of virtualization. Major products get updated:

Out of these four products, only Citrix Xen Server comes with In Memory Read Caching feature.

ESXi has host local DRAM based Caching in two forms.

1. vSAN Client Cache
2. CBRC. It has the added benefit of including deduplication.

Note the IO path for this doesn't require anything strange like going through a DOM0 VM.

@emad-r said in Nested hypervisors?:

Yup go figure for value, Virtualbox has no intention of doing this amazing feature since 2011

Virutalbox is kinda useless as basic things like USB 3 drivers require a 5000$ license.

ESXi has supported nesting longer than all of them, and has a few production use cases for hypervisors on hypervisors.