Another RDS server?
-
@JaredBusch said in Another RDS server?:
@Mario-Jakovina said in Another RDS server?:
We tested speed on our 4 vCPU VMs against Vultr 4 vCPU VM, and ours are much faster (50% if I recall correctly)
That is a waste of a test. It means nothing.
We tested it to compare speed to understand what speed on VM do we get for money.
-
@Mario-Jakovina said in Another RDS server?:
@travisdh1 said in Another RDS server?:
@Mario-Jakovina said in Another RDS server?:
@JaredBusch said in Another RDS server?:
None should have 8 because of his CPU only have 8 cores.
Does it mean that in my case (4-core CPU) VM should use only 3 vCPU even if it is only VM on the host?
Yes, this is correct.
Does it mean than in 1:1 virtualisation, you always "lose" one core?
Ah, I was thinking you had both VMs running on a single host. My bad.
-
@travisdh1 said in Another RDS server?:
@Mario-Jakovina said in Another RDS server?:
@travisdh1 said in Another RDS server?:
@Mario-Jakovina said in Another RDS server?:
@JaredBusch said in Another RDS server?:
None should have 8 because of his CPU only have 8 cores.
Does it mean that in my case (4-core CPU) VM should use only 3 vCPU even if it is only VM on the host?
Yes, this is correct.
Does it mean than in 1:1 virtualisation, you always "lose" one core?
Ah, I was thinking you had both VMs running on a single host. My bad.
@Mario-Jakovina is not the OP. This is a branch conversation.
The OP is @siringo and he has two VM's with 8 vCPU, on the same host, that only has 8 cores.
-
@JaredBusch said in Another RDS server?:
@siringo said in Another RDS server?:
@Pete-S yes, there's one other VM. It has 8vCPU's 16GB of startup RAM and uses dynamic memory.Aside from your runaway program eating memory, this is a problem. You are causing a wait state for CPU availability.
What was drilled into me back when I moved to virtualization is that before the Hypervisor can send the job to the real hardware, enough real cores had to be available to match the number used by the VM.
So each VM must have clock cycles for all 8 real cores available before they can process things.
Wow - I've never heard that before - thanks!
-
@JaredBusch said in Another RDS server?:
@siringo said in Another RDS server?:
@Pete-S yes, there's one other VM. It has 8vCPU's 16GB of startup RAM and uses dynamic memory.Aside from your runaway program eating memory, this is a problem. You are causing a wait state for CPU availability.
What was drilled into me back when I moved to virtualization is that before the Hypervisor can send the job to the real hardware, enough real cores had to be available to match the number used by the VM.
So each VM must have clock cycles for all 8 real cores available before they can process things.
Besides the lack of physical cores (versus vCPUs) causing contention, there is also the problem of 70% CPU.
I believe that the CPU wait time, when the VM is waiting to be scheduled by the hypervisor, is not counted in the guest as CPU usage. You can only see it in the hypervisor as CPU wait time. So if the VM struggles to run, it will not show up as high CPU usage in the VM.
So basically there are several problems. You might solve the contention problem by changing to fewer vCPUs on the VMs, but I don't think you will solve the high CPU usage problem that way. You might even make it worse.
I think you need more cores and more RAM. RAM simply to minimize any swapping to disk when both VMs are asking for memory.
It would be a simple thing to swap the CPU to a 16 core in the same server. A refurb CPU like E5-2683 v4 would set you back about $300. And some more memory. At least as a stop gap measure until you can invest in a proper replacement.
-
Thanks everyone for your useful suggestions.
If I don't answer certain questions, don't take it personally, I appreciate all comments.
So from what I can tell from what others have mentioned, I need to optimise my RAM and sort out my vCPUs (amongst other things).
The server has 3 x 16GB sticks to get to 48GB. I've been given the go ahead for more RAM by the client.
As an aside, situations like this can occur when suppliers don't have stock of parts and clients want solutions in place. There's a chance that all I could get my hands on was 16GB sticks? But then again I should have ordered 4 and not three, but then again maybe all I could get was 3?????????
Anyway ..
So to fix my unbalanced memory problem, I need to add 1 or 5 extra sticks. 1 stick at 16GB would allow me to balance the memory, but I'd like to add more than 16GB.
I can't remember the memory 'layout' but it's usually in groups of two I think????
So I could have 2 groups of 2 x 16GB sticks.
This is what iLO shows me:
As far I can tell I could add an additional 16GB by adding 4 x 4GB sticks, that would give ma an additional 32 GB of RAM in 8 slots which should even up the memory config.
Does that sound correct?
So onto vCPUs.
@JaredBusch you mentioned:
before the Hypervisor can send the job to the real hardware, enough real cores had to be available to match the number used by the VM.If that is the case, then this setup is a real mess.
Over the weekend I'll alter the assignation of vCPUs to VMs.
I think I'll reduce both VMs down to 4 and see how we get on.
With regards to physical CPU's. Would you upgrade the current CPU to something with more cores or add a 2nd CPU?
I think the last time I replaced / installed a CPU it may have been something like a 32032 and had to wash the white paste off my fingers. How hard is it to install CPUs these days? I'm not embarrassed to say that playing with CPUs is something I purposefully stay away from.
Thanks for all the help.
-
@siringo said in Another RDS server?:
I think the last time I replaced / installed a CPU it may have been something like a 32032 and had to wash the white paste off my fingers. How hard is it to install CPUs these days? I'm not embarrassed to say that playing with CPUs is something I purposefully stay away from.
It's actually pretty simple. You need a screw driver, some paper or tissue, some isopropyl alcohol (rubbing alcohol) to clean off the old thermal paste and some new thermal paste. It takes maybe 15 minutes all in all to replace a CPU but give yourself plenty of time. Or have a tech do it for you.
Watch a CPU upgrade in the video below from Dell:
Youtube Video -
@siringo Besides decreasing number of vCPU, I would check does your other (non-RDS) VM needs that much RAM. If you can decrease RAM for second VM, and give fixed amount of RAM to each VM, your RDS VM could significantly decrease swap usage which is probably bottleneck.
SSD would be nice
-
@siringo said in Another RDS server?:
So to fix my unbalanced memory problem, I need to add 1 or 5 extra sticks. 1 stick at 16GB would allow me to balance the memory, but I'd like to add more than 16GB.
I can't remember the memory 'layout' but it's usually in groups of two I think????
So I could have 2 groups of 2 x 16GB sticks.
This is what iLO shows me:As far I can tell I could add an additional 16GB by adding 4 x 4GB sticks, that would give ma an additional 32 GB of RAM in 8 slots which should even up the memory config.
Does that sound correct?The "group" you are talking about is four memory sticks. That's called memory channels because the CPU can read from all four at the same time.
So a E5-2600 v4 series CPU has 4 memory channels and three banks of them for a total of 12 "sticks" (DIMMs) per CPU.
So you're right that you need 4 x 16GB sticks to get to a balanced memory config.
Then you need 4 more if you are going to expand the memory. It can be any size BUT if you intend to add another CPU it has to be the same size. So I suggest 4 x 16GB for the expansion.
-
@siringo said in Another RDS server?:
With regards to physical CPU's. Would you upgrade the current CPU to something with more cores or add a 2nd CPU?
You might have to consider the budget here.
But in general you get better performance with one 16 core CPU than two 8 core CPUs.
It's because one CPU only has access to half the RAM and some of the I/O. So when a CPU need to communicate with something that is connected to the other CPU, the data has to pass through the interconnect between the CPUs. That's a bottleneck and slows it down.
-
@Mario-Jakovina Thanks Mario. I've limited the RAM allocation of this 2nd VM to no more than 16GB. I'll monitor it and see what happens. Thanks.
-
@Pete-S Thanks very much for your help Pete-S, it's greatly appreciated.
I think I'll go and look for a 16 core CPU to replace the current 8 core, depending on price of course.
Thanks once again for all your help.
-
-
When talking about vCPU not add to more physical cores, where do threads come into play?
e.g 12 core cpu (24 threads). So are we saying the VMs running should only have vcpus adding up to less than the physical 12 cores? Or can we go higher due to the 24 threads?
(I'll have a read of the links tomorrow in work, it's Sunday morning and I'm on my phone so don't want to read it on here
) -
@hobbit666 said in Another RDS server?:
When talking about vCPU not add to more physical cores, where do threads come into play?
e.g 12 core cpu (24 threads). So are we saying the VMs running should only have vcpus adding up to less than the physical 12 cores? Or can we go higher due to the 24 threads?
(I'll have a read of the links tomorrow in work, it's Sunday morning and I'm on my phone so don't want to read it on here)Hyperthreads are just two queues for each core. By having two queues the CPU can be a little more effective in executing because one thread can sometimes use resources the other thread is not using.
The hypervisor's CPU scheduler knows this though and tries to make the best out of it. So a 4 vCPU VM will be scheduled to run on 4 different physical cores, as long as it's possible.
The exact algorithm used for scheduling jobs depends on the hypervisor and some hypervisors have different schedulers to pick from as well.
Regardless of that, it's still wise to make vCPU decision based on the number of physical cores per CPU and not logical cores in the system. That will cause the least amount of collisions in the system. But it always a good idea to try different options and see what the results are.
-
@travisdh1 said in Another RDS server?:
Check out
https://www.hpe.com/us/en/insights/articles/10-virtualization-mistakes-everyone-makes-1808.html
and
https://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/
if you want to know why.Looks good reading.
Really need to do more on tweaking on our environment. -
Just remembered this client has a Server sitting idle.
It has a Xeon something, 24GB RAM, Server Std 2016.
I'm thinking, would it be worthwhile building that as a new or 2nd RDS server??? Or moving the RDS VM onto this server?
Changing the RDS server to 4 vCPUs has not made any difference except that I ave a slightly higher CPU load now.
-
@siringo 1. What does your other VM (non-RDS) do, and what OS it uses?
2. What app uses largest part of your CPU load on RDS VM?I would avoid having 2 RDS VMs in your case, I would rather move other VM to this "server sitting idle", and give full resources to RDS VM on main (8-core) server.
-
@travisdh1 said in Another RDS server?:
@Mario-Jakovina said in Another RDS server?:
@JaredBusch said in Another RDS server?:
None should have 8 because of his CPU only have 8 cores.
Does it mean that in my case (4-core CPU) VM should use only 3 vCPU even if it is only VM on the host?
Yes, this is correct.
@JaredBusch said in Another RDS server?:
Basically, everything should be only 1 or 2 vCPU unless specifically noted otherwise.
For DB servers and RDS servers, we use all available cores to better serve more users at the same time. Is something wrong with that?
While you can assign all cores to every virtual machine, it is a horrible idea. The virtual machines will be doing next to nothing but waiting on each other to finish background processes.
Check out
https://www.hpe.com/us/en/insights/articles/10-virtualization-mistakes-everyone-makes-1808.html
and
https://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/
if you want to know why.After reading this articles, I cannot find proof that we have problem with 1:1 virtualization, and giving all physical cores to VM.
Second article is talking about serious overprovisioning (>2vCPU per pCPU) with lots of VMs on single host, and it is not our case.
We have 4vCPU VM on 4-core host, and we do not have issues with CPU speed even on multi-core CPU stress tests. -
Thanks everyone for the help.
I've been speaking with the client and the app that is consuming excess resources will be removed as it's not a business requirement. This should solve the problem.
Once again, thanks to everyone who contributed.
