CloudatCost IOWait Issues
-
Originally where there were issues with CloudatCost performance we were seeing crazy IOWait issues, like caused by trying to power a cloud off of a SAN. Crazy by any measurement. After they did an upgrade we saw storage performance improve a lot. I am now seeing IOWait states (the amount of time that system CPU has to wait on storage) going really high again. In a typical system we would expect to see this far below 1%. Even below .1%. Especially in a mostly idle system. But even on an idle box we are seeing numbers above 15% average IOWait for periods of ten minutes!!
06:00:01 AM CPU %user %nice %system %iowait %steal %idle 06:10:01 AM all 0.31 0.00 0.10 6.75 0.00 92.85 06:20:01 AM all 0.29 0.00 0.10 5.34 0.00 94.27 06:30:01 AM all 0.64 0.00 0.20 9.50 0.00 89.66 06:40:01 AM all 0.01 0.00 0.01 1.38 0.00 98.61 06:50:01 AM all 0.21 0.00 0.07 4.14 0.00 95.58 07:00:01 AM all 1.17 0.00 0.39 16.34 0.00 82.10 07:10:01 AM all 0.94 0.00 0.29 11.78 0.00 86.99 07:20:01 AM all 0.28 0.00 0.10 5.00 0.00 94.62 07:30:01 AM all 0.34 0.00 0.13 4.96 0.00 94.57 07:40:01 AM all 0.61 0.00 0.22 10.71 0.00 88.46 07:50:01 AM all 0.00 0.00 0.01 0.42 0.00 99.56 08:00:01 AM all 0.82 0.00 0.26 12.68 0.00 86.23 08:10:01 AM all 0.11 0.00 0.05 3.42 0.00 96.42 08:20:01 AM all 0.41 0.00 0.13 6.02 0.00 93.43 08:30:01 AM all 0.36 0.00 0.12 5.59 0.00 93.93 08:40:01 AM all 0.01 0.00 0.01 1.76 0.00 98.22 08:50:01 AM all 0.94 0.00 0.29 11.20 0.00 87.58 09:00:01 AM all 0.00 0.00 0.01 1.39 0.00 98.60 09:10:01 AM all 0.93 0.00 0.30 10.99 0.00 87.78 09:20:01 AM all 0.01 0.00 0.01 0.74 0.00 99.24 09:30:01 AM all 0.85 0.00 0.26 9.38 0.00 89.50 09:40:01 AM all 0.12 0.00 0.20 3.91 0.00 95.77 09:50:04 AM all 0.95 0.00 0.35 15.28 0.00 83.41 Average: all 0.42 0.00 0.15 6.69 0.00 92.74 -
These storage numbers are completely crazy. There is a reason that some of the big boys, like Rackspace, use local RAID 10. So much cheaper, so much faster. It's better to have local spinning rust than to have shared SSD that is so overprovisioned.
-
FWIW, not seeing this on either of my boxes, which have little or no traffic.
-
@Danp said:
FWIW, not seeing this on either of my boxes, which have little or no traffic.
What does your SAR report look like? What OS are you running?
-
What would the command be?
-
Here is my SAR output on a lightly used Debian box hosted on Cloud@Cost.
Looks like I'm very far below what you are seeing... this is mostly an idle box... just hosting Wordpress.
-
Here is mine on my Wordpress server on C@C.
02:02:51 AM LINUX RESTART 02:05:01 AM CPU %user %nice %system %iowait %steal %idle 02:15:01 AM all 0.80 0.00 0.25 1.27 0.00 97.68 02:25:01 AM all 0.07 0.00 0.08 0.39 0.00 99.47 02:35:01 AM all 0.06 0.00 0.02 0.15 0.00 99.77 02:45:01 AM all 0.07 0.00 0.03 0.36 0.00 99.55 02:55:01 AM all 0.04 0.00 0.07 0.32 0.00 99.57 03:05:01 AM all 0.06 0.00 0.04 0.39 0.00 99.52 03:15:01 AM all 0.06 0.00 0.02 0.36 0.00 99.56 03:25:01 AM all 0.32 0.00 0.24 5.79 0.00 93.65 03:35:02 AM all 0.06 0.00 0.02 1.24 0.00 98.68 03:45:01 AM all 0.10 0.00 0.03 1.15 0.00 98.72 03:55:01 AM all 0.05 0.00 0.07 1.52 0.00 98.36 04:05:01 AM all 0.06 0.00 0.05 1.31 0.00 98.59 04:15:01 AM all 0.09 0.00 0.10 1.15 0.00 98.66 04:25:02 AM all 0.45 0.00 0.12 0.53 0.00 98.90 04:35:01 AM all 0.06 0.00 0.08 0.53 0.00 99.33 04:45:01 AM all 0.04 0.00 0.02 0.36 0.00 99.58 04:55:01 AM all 0.07 0.00 0.04 0.70 0.00 99.20 05:05:01 AM all 0.07 0.00 0.07 0.35 0.00 99.51 05:15:01 AM all 0.08 0.00 0.07 0.47 0.00 99.38 05:25:01 AM all 0.08 0.00 0.07 0.26 0.00 99.58 05:35:01 AM all 0.76 0.00 0.34 1.76 0.00 97.14 05:45:02 AM all 0.08 0.00 0.02 0.44 0.00 99.46 05:55:01 AM all 0.06 0.00 0.01 0.11 0.00 99.81 06:05:01 AM all 0.08 0.00 0.03 0.53 0.00 99.36 06:15:01 AM all 0.10 0.00 0.02 0.55 0.00 99.33 06:25:01 AM all 0.02 0.00 0.01 0.17 0.00 99.80 06:35:01 AM all 0.08 0.00 0.02 0.63 0.00 99.26 06:45:01 AM all 0.04 0.00 0.02 0.31 0.00 99.63 06:55:01 AM all 0.07 0.00 0.16 0.58 0.00 99.19 07:05:01 AM all 0.05 0.00 0.02 0.10 0.00 99.83 07:15:01 AM all 0.15 0.00 0.04 1.51 0.00 98.30 07:25:02 AM all 0.05 0.00 0.02 0.43 0.00 99.51 07:35:01 AM all 0.07 0.00 0.03 0.47 0.00 99.43 07:45:01 AM all 0.03 0.00 0.02 0.29 0.00 99.66 07:55:01 AM all 1.27 0.00 0.09 2.39 0.00 96.25 08:05:01 AM all 0.07 0.00 0.02 0.51 0.00 99.40 08:15:01 AM all 0.05 0.00 0.02 0.33 0.00 99.61 08:25:01 AM all 0.49 0.00 0.09 0.53 0.00 98.90 08:35:01 AM all 0.04 0.00 0.01 0.29 0.00 99.66 08:45:01 AM all 0.07 0.00 0.02 0.22 0.00 99.69 08:55:01 AM all 0.06 0.00 0.02 0.15 0.00 99.78 09:05:01 AM all 0.07 0.00 0.02 0.19 0.00 99.72 09:15:01 AM all 0.07 0.00 0.01 0.45 0.00 99.47 09:25:01 AM all 0.09 0.00 0.01 0.38 0.00 99.52 09:35:02 AM all 0.06 0.00 0.02 0.27 0.00 99.66 09:45:01 AM all 0.07 0.00 0.02 0.55 0.00 99.36 09:45:01 AM CPU %user %nice %system %iowait %steal %idle 09:55:01 AM all 1.23 0.00 0.08 2.24 0.00 96.44 10:05:01 AM all 0.66 0.00 0.06 1.96 0.00 97.33 10:15:01 AM all 0.05 0.00 0.02 0.45 0.00 99.48 Average: all 0.18 0.00 0.06 0.76 0.00 99.01 -
I'm not seeing anything like that either.
-
-
Mine is higher since having posted. But I know that I am copying some small files during this time...
10:00:01 AM all 0.44 0.00 0.20 16.64 0.00 82.71 10:10:01 AM all 0.59 0.00 0.22 14.60 0.00 84.59 10:20:01 AM all 0.73 0.00 0.26 17.33 0.00 81.68 -
I could try copying some small files from/to this box to see if IOWait is any higher.
-
Here is the average, not spikes, for 24 hours on several boxes.
cc-lnx-jump : 0.22 cc-lnx-ublab : 6.87 cc-lnx-dev1 : 0.17 cc-lnx-rh7lab : 0.28 cc-lnx-rh6lab : 7.20 cc-lnx-mango-st : 0.49 cc-lnx-dblab1 : 0.62 cc-lnx-dblab2 : 2.58 cc-lnx-dblab3 : 0.30 dny-lnx-log : 0.44 dny-lnx-pbx1 : 0.01cc = CloudatCost
dny = Digital Ocean NYC -
I had to install EPEL and SAR - which I thought had been done on my C@C.
Not a good judge of history currently. This is idle.. this box doesn't do anything except occupy space.
10:30:01 AM CPU %user %nice %system %iowait %steal %idle 10:40:01 AM all 0.46 0.00 0.15 11.69 0.00 87.71 Average: all 0.46 0.00 0.15 11.69 0.00 87.71 -
Copied a 500MB file back and forth a few times.
-
@g.jacobse said:
I had to install EPEL and SAR - which I thought had been done on my C@C.
SAR is not a default but should not require the EPEL. No cloud provider, we hope, will have the EPEL by default.
-
@coliver That's a big spike from a little file copying.
-
@scottalanmiller said:
@coliver That's a big spike from a little file copying.
Agreed... I didn't have any spike in views at that point... although that should be minimal.
-
@scottalanmiller
No - I don't think that is the case.... I thought I had it installed.. and it's possible I really did have it installed but have since reimaged the box... thusly it wouldn't run...I had to search back to find your statement about EPEL and SAR to get the correct syntax... then I was being impatient on the reporting.
-
I saw one spike to 11.5% on mine about 11:30AM EST... mine are generally between 2 and 3 %.
I don't have any performance issues on this box at the moment... just doing a bit of piddling with it.
-
Mine was pretty bad last night. I constantly have issues when trying to upload files via sftp or scp anymore (php files usually). At this point I've pretty much given up on CloudatCost. Too bad I can't get a refund for all my Dev and BigDog instances.
