ESXi cluster, advice needed
-
hi, looking for advice on the following exercise:
we have 80 VMs (application servers, databases, web servers etc.) and looking for advice on the architecture and topology of an ESXi cluster, eg how many nodes, storage architecture etc. -
@rtfm said in ESXi cluster, advice needed:
hi, looking for advice on the following exercise:
we have 80 VMs (application servers, databases, web servers etc.) and looking for advice on the architecture and topology of an ESXi cluster, eg how many nodes, storage architecture etc.First of all, what is the business need for ESXi? Depending on the licensing level needed, I can get you setup with a Scale cluster with hardware and all features enabled for the price of licensing from VMWare.
Second, stick to local storage whenever possible, and vsan if shared storage is needed.
At 80 VMs, that's easily a single host today, depending on load, which is what I'd use till a business need caused me to look at more fragile setups.
-
@travisdh1 said in ESXi cluster, advice needed:
@rtfm said in ESXi cluster, advice needed:
hi, looking for advice on the following exercise:
we have 80 VMs (application servers, databases, web servers etc.) and looking for advice on the architecture and topology of an ESXi cluster, eg how many nodes, storage architecture etc.First of all, what is the business need for ESXi? Depending on the licensing level needed, I can get you setup with a Scale cluster with hardware and all features enabled for the price of licensing from VMWare.
Second, stick to local storage whenever possible, and vsan if shared storage is needed.
At 80 VMs, that's easily a single host today, depending on load, which is what I'd use till a business need caused me to look at more fragile setups.
If you don't need HA you can get the VMware Essentials which is something like $700 for 3 years and includes licensing for up to three VMware hosts (6 cpus) and basic support.
If you need HA you get the Essentials Plus which gets you everything you need for 3 hosts. It's much more expensive though.
If you need more than three hosts that's when you need to go with the Standard or higher editions that is when the VMware licensing costs become a big factor. And that's when you pick AMD CPUs because, unless they changed it recently, VMware licenses by the number of CPUs and not cores. So you buy servers with 32 or 64 core AMD Epycs instead of dual CPU Intels.
-
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
That said, at this size you just need one node. Without a specific need for anything more, one node with local storage is all that you would general consider. Storage is always local. You use VSAN or other RLS if you need HA features, but as nothing suggesting HA was in any way useful, we'd assume that it is not since baseline, it is not.
For an average business with an average eighty VM workload, you need one good host with NVMe local storage and you are good to go. Easy, cheap, screaming fast. Take good backups. Recovery is simple.
-
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
-
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
-
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
-
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
-
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
even includes a web based management platform (Cockpit).
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
However you don't have a REST API for KVM (or Hyper-V). That's a big advantage that vSphere/vCenter has. That opens up a multitude of tools to automate everything you're doing.
Terraform does work with libvirt (through a community provider), but because of the way libvirt works your templates and disks are downloaded from where your terraform instance is running (see here: https://github.com/dmacvicar/terraform-provider-libvirt/issues/299). So you're now limited to only running over localhost unless you want to stream your disks to the remote end every time you build a system.
I still love KVM but I'm very sad that it's this many years later and they don't have anything to compete with this.
-
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@travisdh1 said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
I'll echo Travis. For such a small workload, that alone would normally rule out ESXi. ESXi is really only beneficial at massive scale (hundreds or thousands of compute nodes) where you are specifically leveraging unique tech and VMware as your support partner to handle, more or less, everything. Anything smaller than that, I'd recommend ruling them out completely and never entertaining the product. Just doesn't make sense otherwise - high cost, high complexity, high risk, no rewards.
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
I do see, based on much reading here over the years, the benefits of XCP-NG, Hyper-V, and others in a multi-server situation because of the free extra's. Not so much in a single server situation.
You need to stop looking at it from a standpoint of what you already know. Let me ask it another way.
Why are you going to spend that $500 for fewer features than the free alternatives?
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
With that being said, ESXi vs Hyper-V in a large single server situation, It seems setup, stability, small footprint, install on USB/SD Cards are benefits of ESXi. I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
In the words of Dennis Miller, "That is my opinion, I could be wrong."
And I can spin up a new KVM based hypervisor in less than 5 minutes of my time. Plus I save time on the back end by not having to enable basic features for each VM (CPU and RAM dynamic allocation, blanking on what ESXi calls it.) It even includes a web based management platform (Cockpit).
I agree that Hyper-V is such a pain to setup and manage that paying for ESXi could make sense if they were the only two solutions available.
Thanks for the info. Based on most of what I read here, only a few use KVM or XCP-NG in production. I don't have an official poll or anything so I could be mistaken. It just seem if someone wants free, they go Hyper-V.
-
@pmoncho said in ESXi cluster, advice needed:
Question - If a site is only using a single server, I cannot see what benefits other Hypervisors have over a $500 ESXi essentials license other than saving $500? (My knowledge is limited to ESXi and very limited Hyper-V).
Well $500 cheaper. Often faster. More flexibility. Easier to manage. Less to install. No licenses to manage or keep paying for.
Easier, more powerful, cheaper. It's a pretty big improvement.
-
@pmoncho said in ESXi cluster, advice needed:
Not so much in a single server situation.
The same goes both ways. ALL the benefits of ESXi are gone in a single server environment. All. 100% of them. Whereas the benefits of the others are just not as dramatic.
In a tiny environment, all benefits across the board are less. But ESXi gets hit the most, because everything beneficial about the ESXi ecosystem is dependent on a minimum of the Essentials Plus license and multiple machines.
-
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
-
@pmoncho said in ESXi cluster, advice needed:
I feel I can save up to $200 during ESXi setup alone. Plus, $300 for vCenter is an easy bonus for further configuration and management.
No cost savings here, not sure where you are finding that savings. Hyper-V doesn't cost $200 more than ESXi to deploy.
And ESXi is quite expensive once you get to any size. It just keeps costing more and more. And you pay for every little feature.
Bottom line, it costs more, and it doesn't bring benefits. It's not faster, it's not easier, it doesn't do anything beneficial.
You are arguing from the position of "it's not THAT bad", but you haven't said anything that's better, only "not that bad." That you have to use "not that bad" as the position tells us that to you, too, it doesn't feel like a good product, but you aren't ready to articulate it yet. But we are hearing you agree with us because stating that it's only "less bad" is very clear that it's still "worse".
-
@travisdh1 said in ESXi cluster, advice needed:
It even includes a web based management platform (Cockpit).
THis is huge. Whether it's Cockpit for small deployments, virt-manager, or Proxmox or whatever, VMware is so far behind in good management tools.
The majority of our deployments are single server shops, and Vmware is far and away the worst option. Hyper-V is awful compared to Xen and KVM, and still worlds ahead of VMware. The ease of being able to quickly and easily manage these systems remotely is a killer feature in the tiniest of shops.
-
@stacksofplates said in ESXi cluster, advice needed:
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
Not for 80, no. I think he was saying that when you get to the tiny end of the scale, KVM offers advantages in a different way.
-
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
-
@scottalanmiller said in ESXi cluster, advice needed:
@stacksofplates said in ESXi cluster, advice needed:
This is not acceptable for 80 VMs. You can't clone through it. You're going to manually install 80 systems without a base template? Virt-Manager is still the best way to manage.
Not for 80, no. I think he was saying that when you get to the tiny end of the scale, KVM offers advantages in a different way.
Not sure why he would use that case. The case we are talking about is 80 VMs on a single node.
-
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
Not as many, surely. And it varies by the product, but...
- No licensing or license center or logins needed to get downloads and updates (this is a killer feature IMHO, as an MSP we deal with companies that got screwed by this constantly - often when IT pros move on to another company and no one remembers or knows what licenses to manage.)
- Easier remote access, generally without needing to deploy and manage another virtual machine just for that.
- Simpler overall management. VMware is the only platform we are regularly called in to manage because it is "too hard" for the IT teams. We manage everything because people are busy, but VMware is the only one that is regularly "too hard" and people are having issues with it.
- More reliable updates. We see more issues per-VMware install than all other platforms combined. It's not bad, it's just not on par with the market.
- Broader hardware support and options. Software RAID being the massive feature here in the single server space (things like VSANs in the bigger spaces.)
- Included features like built in backups are common in other packages.
- Even in stand alone environments, ability to move workloads to another machine are a huge deal. Stand alone doesn't imply a lack of hardware to send workloads to. It might be that way, but not necessarily.
- Preparation for growth. The single server environment today is the two server environment tomorrow. Flexibility to protect against the unknown always matters.
-
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
@scottalanmiller said in ESXi cluster, advice needed:
@pmoncho said in ESXi cluster, advice needed:
In a single server, It seems the free alternatives lose their feature power and it all comes down to $500 (actually $576).
I have no idea why it seems that way, but it's simply not true. More features, more flexibility... all still there.
In a single server situation, what flexibility or feature exists? I'm only seeing the benefits of free hypervisors kick in in multi-server situations.
Not as many, surely. And it varies by the product, but...
- No licensing or license center or logins needed to get downloads and updates (this is a killer feature IMHO, as an MSP we deal with companies that got screwed by this constantly - often when IT pros move on to another company and no one remembers or knows what licenses to manage.)
- Easier remote access, generally without needing to deploy and manage another virtual machine just for that.
- Simpler overall management. VMware is the only platform we are regularly called in to manage because it is "too hard" for the IT teams. We manage everything because people are busy, but VMware is the only one that is regularly "too hard" and people are having issues with it.
- More reliable updates. We see more issues per-VMware install than all other platforms combined. It's not bad, it's just not on par with the market.
- Broader hardware support and options. Software RAID being the massive feature here in the single server space (things like VSANs in the bigger spaces.)
- Included features like built in backups are common in other packages.
- Even in stand alone environments, ability to move workloads to another machine are a huge deal. Stand alone doesn't imply a lack of hardware to send workloads to. It might be that way, but not necessarily.
- Preparation for growth. The single server environment today is the two server environment tomorrow. Flexibility to protect against the unknown always matters.
Thanks a lot for this. Will keep these in mind.
