Is this normal to see in a VM with multiple CPUs?
-
I increased the threads from 2 to 6
-
@IRJ said:
I increased the threads from 2 to 6
Yup, that's what it looks like. Those are the threads that are doing so much work per thread. Not sure what this application is doing but it is CPU bound. It's possible that the amount of work that it needs to do as a limit and that if you give it enough threading engines that it will be able to keep up and you will see the CPU usage drop. But there is a real possibility that it is not working that way and each "worker" needs 100% of the CPU and no matter how many thread engines you add each one carrying an IIS worker process will be kept at 100% regardless.
Ask the application company what the individual threads are doing here.
-
@scottalanmiller said:
@IRJ said:
I increased the threads from 2 to 6
Yup, that's what it looks like. Those are the threads that are doing so much work per thread. Not sure what this application is doing but it is CPU bound. It's possible that the amount of work that it needs to do as a limit and that if you give it enough threading engines that it will be able to keep up and you will see the CPU usage drop. But there is a real possibility that it is not working that way and each "worker" needs 100% of the CPU and no matter how many thread engines you add each one carrying an IIS worker process will be kept at 100% regardless.
Ask the application company what the individual threads are doing here.
I've have them on the phone and remoted into the server. I will update this thread once I get more info from them
-
My guess is that there is something going on where the code setting up the worker process is caught in a loop or doing something really intense and it does this for every thread. That's a tough one, fixes are either in the code or just require faster cores.... which can be pretty tough to do.
-
This server is the lifeline of our business. This processes electronic loan documents for borrowers. Prospective borrowers are having to deal with much longer waits when applying for loans is bad for business
-
We can still do paper loans, but it takes much longer to do.
-
@IRJ said:
This server is the lifeline of our business. This processes electronic loan documents for borrowers. Prospective borrowers are having to deal with much longer waits when applying for loans is bad for business
That's the kind of software that could easily have very big "setup" processing going on.
-
the CPU usage has been down since the last spike
-
The tech I worked with said he has an idea what is going on, but he needs to talk the developers to confirm.
-
Did they come up with any information for you?
-
@scottalanmiller said:
Did they come up with any information for you?
We are still waiting to hear back.
-
@IRJ that seems like a really long time for their support to figure out something like that.
-
@scottalanmiller said:
@IRJ that seems like a really long time for their support to figure out something like that.
I agree. No spikes over the weekend or any today, though. Its been happening for at least the last 4 weeks. I sent them an email and they said they haven't changed anything yet. I have an after hours call scheduled for tomorrow evening so they can do some DB cleanup
-
Could be a scheduled process to do something like "defrag the database" or "compress something" that runs periodically.
-
@scottalanmiller said:
Could be a scheduled process to do something like "defrag the database" or "compress something" that runs periodically.
Yeah that is what I am thinking because I am noticing a pattern of spikes near 11am, 3pm, and 3 am
-
@IRJ said:
@scottalanmiller said:
Could be a scheduled process to do something like "defrag the database" or "compress something" that runs periodically.
Yeah that is what I am thinking because I am noticing a pattern of spikes near 11am, 3pm, and 3 am
Do you have an incremental backups running at that point? We had some running on an SQL server at points during the day and it would crush the vCPUs of that virtual machine.
-
Good thinking. Backups or snapshots could cause this.
-
@coliver said:
@IRJ said:
@scottalanmiller said:
Could be a scheduled process to do something like "defrag the database" or "compress something" that runs periodically.
Yeah that is what I am thinking because I am noticing a pattern of spikes near 11am, 3pm, and 3 am
Do you have an incremental backups running at that point? We had some running on an SQL server at points during the day and it would crush the vCPUs of that virtual machine.
Unfortunately our Veeam backup runs at 10PM and WSUS updates are done at midnight. So i don't think either of those are causing it.
-
We almost went an entire business day without a spike, but old habits die hard
-
I also managed to grab a quick screenshot of the taskmanager before the peak stopped seconds later
