Hyper-V Guest - Unidentified Network



  • I have a Hyper-V Host server with 4 NICS running Server 2012 R2

    1 NIC is reserved for host management and the other 3 are configured in a NIC Team used exclusively for VMs. This config has been working just fine until this weekend when we had an extended power outage.

    After bringing the host server back online, I restarted the VMs, but now they all get an Unidentified Network error and cannot see the LAN.

    The host server is fine and has full connectivity.

    These VMs are setup with static IPs and this particular LAN does not have a DHCP server.

    My googlefoo has failed me and I cannot find a solution.

    Let me know what other info might help in determining the problem. Any help you all can provide is greatly appreciated.

    Thanks!



  • @pchiodo said in Hyper-V Guest - Unidentified Network:

    I have a Hyper-V Host server with 4 NICS running Server 2012 R2

    1 NIC is reserved for host management and the other 3 are configured in a NIC Team used exclusively for VMs. This config has been working just fine until this weekend when we had an extended power outage.

    After bringing the host server back online, I restarted the VMs, but now they all get an Unidentified Network error and cannot see the LAN.

    The host server is fine and has full connectivity.

    These VMs are setup with static IPs and this particular LAN does not have a DHCP server.

    My googlefoo has failed me and I cannot find a solution.

    Let me know what other info might help in determining the problem. Any help you all can provide is greatly appreciated.

    Thanks!

    Check and make sure the NIC Team and virtual switch is still properly configured?

    Also check the Physical switch this host is connected to.



  • @pchiodo The damned windows magic network detection service needs to die in a damned fire..

    I'll get you a screenshot in a minute.



  • @pchiodo restart this service. Network Location Awareness

    From a GUI, you need to connect remotely, via MMC.

    0_1484677298487_upload-f70951f8-2ef4-41d4-ba4a-865d27604379

    To prevent this in the future, set the service to Automatic (Delayed Start)



  • Sorry, reread your post and the guest VMs are the ones with the problems.. well same scenario..

    The DC was probably a bit slow coming up due to Windows crash handling.

    So because the DC was not fully online, the other VMs did not see the domain network properly. Reboot them or restart the service noted above. I highly recommend setting that damned thing to delayed on all systems.



  • To add to JB's recommendation, you might set the DC to boot a few mins before the others allowing it time to come up fully before the rest.



  • Yeah... I've tried all of the above, and the VMs still come up with Unidentified Network. I have a number of other HV Hosts configured the exact same way, and none of these have this problem.

    I've rebooted the host several times, and had already changed NLA to delayed start on both the host and the guests.

    At this point, I'm dissolving the team and removing the virtual switch, I'll then recreate the team and the virtual switch and reattach the guests, and see where I'm at.



  • Oh, and I have 2 DCs, so it should see one of them almost immediately



  • @pchiodo said in Hyper-V Guest - Unidentified Network:

    Yeah... I've tried all of the above, and the VMs still come up with Unidentified Network. I have a number of other HV Hosts configured the exact same way, and none of these have this problem.

    I've rebooted the host several times, and had already changed NLA to delayed start on both the host and the guests.

    At this point, I'm dissolving the team and removing the virtual switch, I'll then recreate the team and the virtual switch and reattach the guests, and see where I'm at.

    Well, that didn't pan out. Same problem...



  • Would you happen to have your DHCP server on that same HOST? Is the HOST set to a static or dynamic ip?



  • @Tim_G Ha, I completely missed these two lines:

    "The host server is fine and has full connectivity.
    These VMs are setup with static IPs and this particular LAN does not have a DHCP server."

    Ignore my above post.



  • OK, so now I have dissolved the team again, removed the virtual switch, configured one of the physical adapters with a static IP and then have created a new virtual switch with only the one NIC. So as it stands, the management NIC is NIC1 and the Virtual switch NIC is NIC2.

    When I set the static on NIC2, it immediately connected to the network as normal.

    As soon as I recreated the virtual switch, NIC 2 on the physical box changed to Unidentified network, and of course, the VM has the same issue.

    So when NIC2 bound the host to the virtual, something went haywire.

    Still struggling..... ugh



  • As soon as I remove the virtual switch, NIC2 comes back online, and can be pinged from the network. It appears this issue is on the host level.



  • Is it a Broadcom NIC? Is the firmware up to date? There are issues that are now resolved with Broadcom NICs if you update the firmware, if not, (on the host) you'll need to go in to the advanced properties of the NIC and disable Virtual Machine Queues:

    Set-NetAdapterAdvancedProperty NIC1 -DisplayName “Virtual Machine Queues” -DisplayValue Disabled
    Set-NetAdapterAdvancedProperty NIC2 -DisplayName “Virtual Machine Queues” -DisplayValue Disabled
    Set-NetAdapterAdvancedProperty NIC3 -DisplayName “Virtual Machine Queues” -DisplayValue Disabled
    Set-NetAdapterAdvancedProperty NIC4 -DisplayName “Virtual Machine Queues” -DisplayValue Disabled

    If the above is all good or doesn't apply to you, have you:

    1. Set up the 3-NIC team FIRST
    2. Assign a static IP to the team
    3. Create a vSwitch in Hyper-V Manager, select the team NIC, UN-CHECK "Allow management operating system to share this network adapter".
    4. Assign above vSwitch (in #3 above) as the "Virtual Switch:" in your virtual machine's settings. You can reassign a VM's vNIC while it's running.


  • Disabling VMQ is much simpler and should always be disabled with Broadcom NICs.

    Get-NetAdapterVmq | Disable-NetAdapterVmq 
    

    https://mangolassi.it/topic/8358/i-hate-vmq



  • VMQ has already been disabled. That was a standard when I loaded these boxes.



  • What was the last patch installed on the hypervisor?



  • @JaredBusch said in Hyper-V Guest - Unidentified Network:

    Disabling VMQ is much simpler and should always be disabled with Broadcom NICs.

    Get-NetAdapterVmq | Disable-NetAdapterVmq 
    

    https://mangolassi.it/topic/8358/i-hate-vmq

    Generally, you DO want to have VMQ enabled on your VM's Network Adapter, under Hardware Acceleration: "Enable virtual machine queue", and enabled on the HOST's network adapter. VMQ is not the issue. With VMQ enabled, you get even better performance (well, at least with 10gb speeds... probably not noticeable with 1gb). With VMQ disabled, the network load is on the Hosts CPU. VMQ off-loads it to the NIC, where it should be.

    Where the problem lies, is with a select number of Broadcom NICs... specifically with the 57xx based chipsets. If you have one of the effected NICs, you should update the drivers, as it has now been resolved. If you can't update, then disable VMQ.

    Here's the latest official Microsoft article regarding it:

    https://support.microsoft.com/en-us/kb/2986895



  • I'm back to this problem. I moved the load from this server to another server so I could further investigate. I think I found the issue, but not a resolution.

    In Device Manager, I still have two rogue entries:

    Hyper-V Virtual Ethernet Adapter
    Microsoft Hyper-V Network Switch Default Miniport

    Both of these have the following error:

    This device is not working properly because Windows cannot load the drivers required for this device. (Code 31)

    An object ID was not found in the file

    In PS the Get-VMNetworkAdapter - ManagementOS and Get-VMSwitch both come back with no results, yet they are listed in Device Manager. I can disable them in DM, but it does not resolve the issue. DM does not let me remove them, and Remove-VMSwitch and Remove-VMNetworkAdapter have no effect.

    Any ideas?



  • @pchiodo said in Hyper-V Guest - Unidentified Network:

    I'm back to this problem. I moved the load from this server to another server so I could further investigate. I think I found the issue, but not a resolution.

    In Device Manager, I still have two rogue entries:

    Hyper-V Virtual Ethernet Adapter
    Microsoft Hyper-V Network Switch Default Miniport

    Both of these have the following error:

    This device is not working properly because Windows cannot load the drivers required for this device. (Code 31)

    An object ID was not found in the file

    In PS the Get-VMNetworkAdapter - ManagementOS and Get-VMSwitch both come back with no results, yet they are listed in Device Manager. I can disable them in DM, but it does not resolve the issue. DM does not let me remove them, and Remove-VMSwitch and Remove-VMNetworkAdapter have no effect.

    Any ideas?

    Reinstall Hyper-V Server. Something is hosed. Well I guess in your case it is Server 2012 DataCenter?



  • @JaredBusch That's really using a sledgehammer when a scalpel should be sufficient. I'd really like to find a resolution in case this comes up again. nuke and reload is a last resort IMHO.



  • @pchiodo said in Hyper-V Guest - Unidentified Network:

    @JaredBusch That's really using a sledgehammer when a scalpel should be sufficient. I'd really like to find a resolution in case this comes up again. nuke and reload is a last resort IMHO.

    Thanks for posting this - and while your hopefully found solution is a bit more useful considering that 2012 is still supported and sorta current, my wanting to fix my booting 2003 issue is along the same guise.

    Sadly for both of us, it might not be worth the expense to our companies (in our time not working on other issues, if not in direct billing) versus just wiping and reinstalling.



  • @Dashrender I totally agree, and if this was a required production server, I'd be right on board. This particular server is not required for production, and I have moved its load to another Hyper-V server. Yes, I would like it back in the mix for load balancing and as a backup, but I still would like to take a little time to try and nail down the exact issue and solution before I just wipe it and reload.



  • @pchiodo said in Hyper-V Guest - Unidentified Network:

    @Dashrender I totally agree, and if this was a required production server, I'd be right on board. This particular server is not required for production, and I have moved its load to another Hyper-V server. Yes, I would like it back in the mix for load balancing and as a backup, but I still would like to take a little time to try and nail down the exact issue and solution before I just wipe it and reload.

    Right - but you missed the part about how this is possibly not beneficial to your company.

    Like you, my 2003 server isn't in production either, I'm trying to make a CYA fully working/tested backup to put into the safety deposit box. Fixing the boot problem fully doesn't really help my company in any way since I have a usable solution for the boot issue. Fixing it takes me away from doing something of real value for my company, the same goes for you and solving this issue.

    In the end, you and I both personally gain by having additional knowledge we didn't have before, but what we don't know is if that knowledge will have any outward value in the future. While your situation has a higher likeliness of having future value, people are still deploying Hyper-V 2012, and it's likely a very similar setup probably exists in Hyper-V 2016. My situation is completely different in that 2003 is old, does booting in a completely old manner - using P2V that's a mostly dead situation (very little P2Ving done anymore, most of it was already done), so I have very little chance of ever using this knowledge in the future.



  • @Dashrender said in Hyper-V Guest - Unidentified Network:

    @pchiodo said in Hyper-V Guest - Unidentified Network:

    @Dashrender I totally agree, and if this was a required production server, I'd be right on board. This particular server is not required for production, and I have moved its load to another Hyper-V server. Yes, I would like it back in the mix for load balancing and as a backup, but I still would like to take a little time to try and nail down the exact issue and solution before I just wipe it and reload.

    Right - but you missed the part about how this is possibly not beneficial to your company.

    Like you, my 2003 server isn't in production either, I'm trying to make a CYA fully working/tested backup to put into the safety deposit box. Fixing the boot problem fully doesn't really help my company in any way since I have a usable solution for the boot issue. Fixing it takes me away from doing something of real value for my company, the same goes for you and solving this issue.

    In the end, you and I both personally gain by having additional knowledge we didn't have before, but what we don't know is if that knowledge will have any outward value in the future. While your situation has a higher likeliness of having future value, people are still deploying Hyper-V 2012, and it's likely a very similar setup probably exists in Hyper-V 2016. My situation is completely different in that 2003 is old, does booting in a completely old manner - using P2V that's a mostly dead situation (very little P2Ving done anymore, most of it was already done), so I have very little chance of ever using this knowledge in the future.

    Agreed! In my particular case, I can see this happening again. We will likely not move off Server 2012 for the next 2-3 years depending on how fast our 3rd party apps adopt server 2016. This being the case, having a solution that takes minutes vs hours would be very valuable.



  • @pchiodo Your situation sounds similar to an issue I had a while back. I could not figure out what cause was but my workaround was to disable and then enable the network card in the VM. I use a schedule task to run at start up. Below is the script I use.

    netsh interface set interface "Ethernet 2" DISABLED
    netsh interface set interface "Ethernet 2" ENABLED



  • Obviously use your Ethernet adapter's name



  • @syko24 said in Hyper-V Guest - Unidentified Network:

    @pchiodo Your situation sounds similar to an issue I had a while back. I could not figure out what cause was but my workaround was to disable and then enable the network card in the VM. I use a schedule task to run at start up. Below is the script I use.

    netsh interface set interface "Ethernet 2" DISABLED
    netsh interface set interface "Ethernet 2" ENABLED

    He is not dealing at the VM level right now. He is on the host with no VM's



  • Sorry missed the part where you said you moved the VMs to another system.

    Here is another possible solution. - http://www.wintips.org/fix-network-adapter-code-31-device-is-not-working-properly/

    1. Go into the registry (On the host) and export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network
    2. Delete the subkey Config if it exists
      • If Config doesn't exist, look at the keys below Network and identify the rogue nics and delete the keys
    3. Go into device manager and try to uninstall the rogue nics
    4. Reboot


  • The solution to removing them and re-establishing the vSwitch and VM Adapters was to remove the Hyper-V Role and then reinstall the role.
    Specifically the following steps:

    1. Remove the network connections to the VMs
    2. Remove the vSwitch
    3. Dissolve the NIC Team
    4. Remove the Hyper-V Role
    5. Reboot
    6. Create new NIC Team
    7. Reinstall Hyper-V Role (Create vSwitch during role installation)
    8. Reboot
    9. Reattach the VMs and assign IPs if necessary

    Everything appears to work as normal now. Unsure of what caused the issue, but may have been related to a hard shutdown caused by a power outage and failure of the backup generator.


Log in to reply