XenServer Issues with SMB over PV NIC
-
Did you have these issues prior to updating the host(s)?
-
Yes with the VM that would not copy at a reasonable rate. That is why I moved to 6.5SP1 from 6.2 base. It seemed like everything was ok when we first installed but over time (I am guessing with windows updates) it just kept getting slower and slower. I traced thru everything I could think of from an IO perspective moved the hosts from RAID 6 to RAID 10, on the VMs upped the vCPU counts, and upped the RAM. When none of that worked I purchased Citrix support and they came up with the NIC offloading.
-
Have you attempted a 2008R2 server as a clean install and see if these issues persist?
You mentioned that you upgraded from 6.2 to 6.5SP1 because of these issues, but these issues could have been caused because of the 6.2 PV drivers.
-
@jshiers said:
Yes with the VM that would not copy at a reasonable rate. That is why I moved to 6.5SP1 from 6.2 base. It seemed like everything was ok when we first installed but over time (I am guessing with windows updates) it just kept getting slower and slower. I traced thru everything I could think of from an IO perspective moved the hosts from RAID 6 to RAID 10, on the VMs upped the vCPU counts, and upped the RAM. When none of that worked I purchased Citrix support and they came up with the NIC offloading.
If it is tied to a Windows update that means it is less likely to be driver related.
-
What happens if you switch that 2008R2 VM from a PV NIC to a Non-PV Nic? Can you do that in XenServer?
-
I can't prove the performance issue is tied to Windows updates. What I can prove is how the VMs are preforming today and relate that back to the fact that 6-12 months ago I didn't seem to have these issues with the same software packages. Performance started going down gradually as we didn't really notice issues until it "suddenly" occurred to us that things were not running well. In the overnight I have spun up a 2012R2 VM to do some testing with. I am also wondering if I need to flash the BIOS on the underlying hardware as if the software layer was updated and is expecting the NIC to perform a specific way but the BIOS isn't doing that then maybe my issue is there?
-
@jshiers the BIOS would effect every system you have, so I would update (to update) but don't expect there to be a performance change. As your other VM's aren't have any issues.
With your 2012R2 VM monitor it's performance and see if there is a difference. It's not Apples to Apples, as this issue is occurring with your 2008R2 server.
Do you have a VM backup that you can light up of the 2008R2 server from several months back and see if the issue persist?
-
@dafyre You'd have to install dedicated hardware drivers to the VM rather than the xentool.iso drivers.
Which I'm not certain what would happen with it in this case.
Additionally I don't think it would address the issue. The VM changed, and performance was degraded.
Speaking of, how much free space does the vHDD have on it? I've seen windows servers come to a crawl when the drive is full. Worth a look.
-
I have made sure all the VMs have at least 20 gig of free vHDD space to eliminate any issue with drive space. We also have used a Fluke MicroScanner to check out our physical cables and found a couple of suspect cables that we replaced as well.
Here are some test results, SMB file copy from a 2008R2 VM that has NIC offloading turned on to the new 2012R2 box with no PV tools so its using the base Windows Realtek driver and got 24 Mbps. Not great but much better than the 64-72 Kbps I get between two 2008R2 boxes with offloading turned on. Same test from 2008R2 VM with NIC offloading turned off and I get the same 24 Mbps. All of these VMs are on the same host.
Ok installed the PV tools on the 2012R2 server. Copy from a 2008R2 VM with NIC Offloading ON and the 350 meg file copied in a flash. Resource Monitor showed 2 Gbps. Copy from the VM with NIC Offloading OFF and the copied at 225 Mbps.
Maybe I should just start do inplace upgrades on my servers!!!!
-
Upgrading to the newest release of the Server OS never really hurts, so long as your systems are capable of running on that OS.
I'm glad you were able to remove trouble patch cables, and your testing with 2012R2 seems to have resolved the issue.
Are you able to perform an inline upgrade in a test environment so you're not effecting your production systems?
-
@jshiers said:
I can't prove the performance issue is tied to Windows updates. What I can prove is how the VMs are preforming today and relate that back to the fact that 6-12 months ago I didn't seem to have these issues with the same software packages. Performance started going down gradually as we didn't really notice issues until it "suddenly" occurred to us that things were not running well. In the overnight I have spun up a 2012R2 VM to do some testing with. I am also wondering if I need to flash the BIOS on the underlying hardware as if the software layer was updated and is expecting the NIC to perform a specific way but the BIOS isn't doing that then maybe my issue is there?
Unlikely that you need to flash the BIOS but that does not mean that it is not a good idea, but unlikely to help.
That performance has gone down slowly means that drivers are also unlikely to be involved.
-
@jshiers said:
Maybe I should just start do inplace upgrades on my servers!!!!
That's an option for sure. Snap them, upgrade, test and see.
