Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
I have limited experience with Hyper-v
Where is the storage? Local? iscsi?
Is the host and guest on the same network? Is the router getting overwhelmed? Does the router even get traffic when this happens?
what does the switch say? You should be able to log in and check its realtime log.You just beat me to it. Correct, everything is local storage.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
The only reasonable answer here was if there was no change, to have it disabled so the switch and systems operate in the most common fashion.
I'd recommend going and disabling it.
If jumbo frames was the issue, I'd expect the file transfer to the host to also be affected but it's not. That's performing really, really well. It's only to the VM where the issue is seen.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also generally speaking you don't want to enable jumbo frames without reason as it can lead to weird issues. Do you know what reasoning there was / is to have this change in place?
I've made the recommendation to try turning off jumbo frames when I first took the client on and they did a few tests with no changes in performance so rather than change every PC and server and switch, they decided to keep it on (they had only tested turning it off to see if there was a difference).
The only reasonable answer here was if there was no change, to have it disabled so the switch and systems operate in the most common fashion.
I'd recommend going and disabling it.
If jumbo frames was the issue, I'd expect the file transfer to the host to also be affected but it's not. That's performing really, really well. It's only to the VM where the issue is seen.
Has this issue been tested with JF disable, or was JF disabled just to see what would happen.
Generally unless you have a good reason, just leave the default settings alone. Is the rule of thumb to go by.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
When I run ping -t tests from the workstation to the host, replies are consistently <1ms or 1ms and 0% loss. Same ping -t tests to the file server VM result in <1ms to as high as 11ms with 0% loss.
11ms is high for a local ping.
From the workstation
try a tracert to host
try tracert to vmFrom host
do a tracert to vm
do a tracert to workstationHow is the vnic on the guest setup?
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
When I run ping -t tests from the workstation to the host, replies are consistently <1ms or 1ms and 0% loss. Same ping -t tests to the file server VM result in <1ms to as high as 11ms with 0% loss.
11ms is high for a local ping.
From the workstation
try a tracert to host
try tracert to vmFrom host
do a tracert to vm
do a tracert to workstationHow is the vnic on the guest setup?
All results look good.
Workstation to VM Guest = 1 1ms <1ms <1ms FILE-SERV [192.168.100.83] Workstation to Host = 1 1ms <1ms <1ms vHOST [192.168.100.232] VM Guest to Workstation = 1 1ms <1ms <1ms VID-STATION [192.168.100.71] Host to Workstation = 1 1ms <1ms <1ms VID-STATION [192.168.100.71]The vNic is a simple external network. Both workstation and host physical NICs are Intel X520-DA1 (drivers are current).
-
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
-
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
edit: whoops i forgot xfer to host is fast -
@minion said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@momurda said in [Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD]
11ms is high for a local ping.
Very high. Everything here is less then 1ms.
Agreed, which is why I'm leaning towards there being something up with the VM, especially since transfers to and from the host are perfect. At a loss for where to look next as I've eliminated the usual suspects.
-
@momurda said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Hmm, well that looks fine.
All the machines on same network.Perhaps the storage cannot keep up with 10G (or 5G). What is the local storage? Spinning raid 10? Check the 2012 file server event viewer for storage timeout/retries.
The storage can definitely handle the 10G. As I mentioned earlier, I'm easily able to reach transfer speeds of 600MBps to and from the host. It is configured as RAID10 nearline 10TB Seagate drives.
-
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
-
@DustinB3403 said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Are the drivers up to date inside of the VM? I know you said that they are for the nic but it was a bit vague and could've been only for the hypervisor and not the VM its self.
All PCs and servers and VMs are fully up to date. If there is one thing they do well at this place is patch/update often and early.
-
I'd go back to looking at the NICs. In Hyper-V the management console can have one physical NIC, and the VMs another physical NIC. Ensure that VMQ is disabled as @dafyre mentioned, and check the configuration of the VM NIC. You may want to try swapping the NIC assignments and then try your tests again.
If teaming is enabled for the VM NICs, I would check this config, and possibly dissolve the team to test.
-
Checking the vmqueue first.
Then try benchmarking disks on host and vm with any free tool just to understand if this is a disk issue in the vm or a networking one -
How about the physical port on the switch? change where the VM host is plugged in, test, then change the PC and test.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Also check that VMQ is disabled on your NICS.
Check out the Disable-NetAdapterVMQ powershell commands.
So much this. Check all pieces. The above command can disable it on the host, but the guest also has a setting that should be disabled.
-
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
-
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
-
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
-
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
-
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@dafyre said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@Tim_G said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@NashBrydges said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
@JaredBusch said in Transfer Large File To Host = GOOD Transfer Large File To VM File Server On Same Host = BAD:
Disable-NetAdapterVMQ
Yeah I was aware of the VMQ issues so that was disabled.
Once the office had closed for the day and after an hour of toying with everything I could find, I decided to completely uninstall the network drivers and reinstall from scratch. Deleted the vswitch and recreated it. Also uninstalled network drivers from workstation and reinstalled. Rebooted everything including the switch and it looks like everything is back to normal...for now. Not sure why this changed anything since none of the settings have changed. Just bizarre.
Thanks for your help everyone.
You do not want to disable VMQ on 10Gb NICs. That's what it is meant for. The issue was that VMQ was being enabled by default for 1Gb adapters, and that was causing the problems.
You disable VMQ on anything that is Intel or Broadcomm period. Their drivers are seriously bugged for that. I've got 4 Server 2012 R2 bare metal machines with Intel and Broadcomm 10 gig NICs in them. Network performance absolutely sucks until I disable VMQ.
Up to date firmware on the NICs?
My experience has been the opposite, as well as my research on the subject.
VMQ on 10Gb NICs is fine, that's what it was designed for. VMQ on 1Gb NICs is not fine.
When I disable VMQ on a busy 10Gb NIC on a Hyper-V host, performance goes down.
Yepp. Firmware is up to date (did an update about 2 months ago, I think). The systems have been operational for a year or better.
It's all Cisco gear on the networking side. I've read the same thing that you have that says 10gig is fine to enable VMQ, but on Intel & Broadcomm NICs, it is not.
