Poor network bandwidth on VM (failover cluster)



  • This is a separate thread to a thread on the same server but different problem. http://mangolassi.it/topic/7368/hyper-v-failover-live-migration-failed-error-21502

    Currently, everything seems to be working fine... at least on the hypervisor side. The VM server is meant for file server and it need to perform in its top notch performance during business hours. However, the performance is barely acceptable. When saving a blank file (excel) on to the VM it takes couple of seconds with a progress bar. This action should be instant. What can I do to improve the file server performance?



  • This could be related to the disk performance and not the network performance.

    Just because the document is being written to a network share doesn't mean that is the issue.


  • Service Provider

    Before I go digging... what is the storage situation? Local storage?



  • @DustinB3403 said:

    This could be related to the disk performance and not the network performance.

    Just because the document is being written to a network share doesn't mean that is the issue.

    How is this related to disk performance? Please enlighten me.

    @scottalanmiller said:

    Before I go digging... what is the storage situation? Local storage?

    Not sure what you mean but it is configured with vSAN.


  • Service Provider

    @LAH3385 said:

    Not sure what you mean but it is configured with vSAN.

    That's what I wanted to know.


  • Service Provider

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?



  • @scottalanmiller said:

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

    2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth



  • When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

    If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

    I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.



  • @LAH3385 said:

    @scottalanmiller said:

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

    2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

    In the case of RDP - what do the system resources utilization look like?



  • @Dashrender said:

    When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

    If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

    I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

    How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

    @Dashrender said:

    @LAH3385 said:

    @scottalanmiller said:

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

    2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

    In the case of RDP - what do the system resources utilization look like?

    resources utilization on the my PC, hypervisor, or VM?



  • @LAH3385 said:

    @Dashrender said:

    In the case of RDP - what do the system resources utilization look like?

    resources utilization on the my PC, hypervisor, or VM?

    That would depend, are you RDPing into the hypervisor or VM? in either case the hypervisor could be at fault. I'd start by looking at the hypervisor. I know in VMWare you can see the resource utilization of all on a chart, so you should be able to see everything you need from within the hypervisor I would guess.


  • Service Provider

    @LAH3385 said:

    @scottalanmiller said:

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

    2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

    Could be. Does the image update quickly?



  • @LAH3385 said:

    @Dashrender said:

    When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

    If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

    I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

    How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

    Starwind console should have graphs available for all kinds of resources utilisation. Btw, you don't need crossover cable on 1Gbit and faster ethernet cards.



  • @Dashrender said:

    @LAH3385 said:

    @Dashrender said:

    In the case of RDP - what do the system resources utilization look like?

    resources utilization on the my PC, hypervisor, or VM?

    That would depend, are you RDPing into the hypervisor or VM? in either case the hypervisor could be at fault. I'd start by looking at the hypervisor. I know in VMWare you can see the resource utilization of all on a chart, so you should be able to see everything you need from within the hypervisor I would guess.

    I just gave the VM more virtual processor from 1 to 4. It does seem more responsive while RDP. But the progress bar when saving still there. The graph on Performance (Task Manager) does not seem to be indicating anything out of ordinary. The graph goes up and down like a heart beat on cross over ethernet and wobbling on internet.

    @scottalanmiller said:

    @LAH3385 said:

    @scottalanmiller said:

    So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

    2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

    Could be. Does the image update quickly?

    How do I check? It should be within couple of seconds.. or every second.

    @marcinozga said:

    @LAH3385 said:

    @Dashrender said:

    When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

    If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

    I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

    How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

    Starwind console should have graphs available for all kinds of resources utilisation. Btw, you don't need crossover cable on 1Gbit and faster ethernet cards.

    I couldn't find where the graph would be located at. But this is the setting on Synchronization priority
    0_1452184423155_upload-864c3478-4cc1-418f-9abe-4a38e2d7b6c9



  • What antivirus (if any) is running on that server? What happens if you disable it?



  • @marcinozga said:

    What antivirus (if any) is running on that server? What happens if you disable it?

    No AntiVirus. If any.. it would be w/e built-in into Windows server 2K12 R2



  • You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.



  • @dafyre said:

    You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

    I moved it to 9/10 client access. Very little to no different.



  • @LAH3385 said:

    @dafyre said:

    You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

    I moved it to 9/10 client access. Very little to no different.

    Might be a wise thing to set it back closer to the defaults.

    Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

    0_1452192335353_upload-f8415521-98e9-4429-bd11-eeb1e68fdad2



  • @dafyre said:

    @LAH3385 said:

    @dafyre said:

    You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

    I moved it to 9/10 client access. Very little to no different.

    Might be a wise thing to set it back closer to the defaults.

    Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

    0_1452192335353_upload-f8415521-98e9-4429-bd11-eeb1e68fdad2

    What am I looking for? How long should I run the test?



  • set the counters, then move some files around, then take a screen shot and post.



  • @Dashrender said:

    set the counters, then move some files around, then take a screen shot and post.

    Idle 1min
    0_1452193514171_upload-0630f92b-5ad0-47d2-b048-23ce8b69677c
    File Transfer 11 minutes
    0_1452193504024_upload-951c5e09-5883-406b-8533-0d2c0d029909



  • @LAH3385

    These are some tests I conduct.
    visor1
    visor2 (the host where VM resides at the moment)
    VM
    File size 1.8GB (contain 450 files of 4MB each)

    visor1 to visor2
    0_1452194713907_upload-ff1e7c09-796d-406f-a189-b6bb37bc723d
    visor2 to visor1
    0_1452194777230_upload-5180d670-c663-4f04-856d-30e38f833b4a
    visor1 to VM
    0_1452195144939_upload-cf2436ed-6db6-4eec-b145-535bd5c566bf
    visor2 to VM
    0_1452195249146_upload-a93df16d-4510-4eec-9748-b14281cec9c1

    Connection from my PC to visor1/2 or VM is around 6MB/s to 11MB/s (average around 7MB/s)



  • @LAH3385 said:

    @dafyre said:

    @LAH3385 said:

    @dafyre said:

    You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

    I moved it to 9/10 client access. Very little to no different.

    Might be a wise thing to set it back closer to the defaults.

    Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

    0_1452192335353_upload-f8415521-98e9-4429-bd11-eeb1e68fdad2

    What am I looking for? How long should I run the test?

    I saw you post your permon screen... Look under the disk performance tabs and see what you are getting?



  • @dafyre

    Weird thing is only hypervisor1 able to view report. visor2 and VM return with error
    0_1452201737590_upload-fef9d8b5-b0a5-4768-9994-2f5eccc051a5
    I really do not think my disk is the bottle neck here.
    I did found a setting on GPO [Network/Background Intelligent Transfer Service (BITS)] It should not have impact on Foreground transfer.


  • Vendor

    @DustinB3403 said:

    This could be related to the disk performance and not the network performance.

    Just because the document is being written to a network share doesn't mean that is the issue.

    +1


  • Vendor

    Starwind console should have graphs available for all kinds of resources utilisation. Btw, you don't need crossover cable on 1Gbit and faster ethernet cards.

    I couldn't find where the graph would be located at. But this is the setting on Synchronization priority
    0_1452184423155_upload-864c3478-4cc1-418f-9abe-4a38e2d7b6c9

    It's on the Performance tab. But you can check the Windows performance monitor as well.
    As about the priority - I'd recommend you to keep it in the middle. It relates to the synchronization after failures (FastSync or FullSync)


  • Vendor

    @LAH3385 said:

    @LAH3385

    These are some tests I conduct.
    visor1
    visor2 (the host where VM resides at the moment)
    VM
    File size 1.8GB (contain 450 files of 4MB each)

    visor1 to visor2
    0_1452194713907_upload-ff1e7c09-796d-406f-a189-b6bb37bc723d
    visor2 to visor1
    0_1452194777230_upload-5180d670-c663-4f04-856d-30e38f833b4a
    visor1 to VM
    0_1452195144939_upload-cf2436ed-6db6-4eec-b145-535bd5c566bf
    visor2 to VM
    0_1452195249146_upload-a93df16d-4510-4eec-9748-b14281cec9c1

    Connection from my PC to visor1/2 or VM is around 6MB/s to 11MB/s (average around 7MB/s)

    Actually measuring the performance with the file copy is not the best way
    http://blogs.technet.com/b/josebda/archive/2014/08/18/using-file-copy-to-measure-storage-performance-why-it-s-not-a-good-idea-and-what-you-should-do-instead.aspx

    I'd recommend you to run the IOmeter benchmark against StarWind RAM disk through the networks. It should show the real numbers


  • Vendor

    @LAH3385 BTW, as I mentioned in the other post to you, we are welcomed to jump with you on the remote session to look deeper into the issue and try to solve it. I'm going to PM you right now.


Log in to reply
 

Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.