Intermittent call quality issues with a FreePBX instance on Vultr in Chicago



  • Having some intermittent problems with an instance in Chicago.

    4 office locations with phones. (3 fiber, one coax)
    3 residences with phones. (1 fiber, 1 coax, 1 copper)
    4 PJSIP /SIP trunks to 4 VoIP.ms POPS (chicago2, chicago3, chicago4, denver2)
    1 IAX2 trunk to my PBX (also in Chicago datacenter)

    Users at all 4 offices and one of the homes experience intermittent bad call quality where the person on the PBX cannot hear most of what the other party says like missing every other second of the audio. But the other person can hear the PBX user perfectly the entire time.

    When it happens, it seems to happen at all sites at the same time. It is not tied to only one site. This leads me to suspect Vultr or VoIP.ms

    [[email protected] ~]# sar
    Linux 2.6.32-642.6.2.el6.x86_64 (bpbx.flstl.com) 	08/18/2017 	_x86_64_	(1 CPU)
    
    12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    12:20:01 PM     all      7.08      0.00      3.60      0.04      0.32     88.96
    12:30:01 PM     all      6.00      0.00      2.70      0.06      0.24     91.00
    12:40:01 PM     all      6.26      0.00      2.93      0.05      0.20     90.57
    12:50:01 PM     all      7.90      0.00      4.13      0.06      0.25     87.67
    01:00:02 PM     all      8.18      0.00      4.56      0.05      0.30     86.92
    Average:        all      6.08      0.00      3.02      0.12      0.20     90.57
    

    response time to VoIP.ms sites

    [[email protected] ~]# ping denver2.voip.ms
    PING denver2.voip.ms (173.248.159.210) 56(84) bytes of data.
    64 bytes from 173.248.159.210: icmp_seq=1 ttl=51 time=23.2 ms
    64 bytes from 173.248.159.210: icmp_seq=2 ttl=51 time=23.1 ms
    64 bytes from 173.248.159.210: icmp_seq=3 ttl=51 time=23.3 ms
    ^C
    --- denver2.voip.ms ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2878ms
    rtt min/avg/max/mdev = 23.188/23.251/23.346/0.141 ms
    [[email protected] ~]# ping chicago2.voip.ms
    PING chicago2.voip.ms (208.100.39.53) 56(84) bytes of data.
    64 bytes from 208.100.39.53: icmp_seq=1 ttl=57 time=1.18 ms
    64 bytes from 208.100.39.53: icmp_seq=2 ttl=57 time=1.12 ms
    ^C
    --- chicago2.voip.ms ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1990ms
    rtt min/avg/max/mdev = 1.129/1.157/1.186/0.044 ms
    [[email protected] ~]# ping chicago3.voip.ms
    PING chicago3.voip.ms (208.100.39.54) 56(84) bytes of data.
    64 bytes from 208.100.39.54: icmp_seq=1 ttl=57 time=1.13 ms
    64 bytes from 208.100.39.54: icmp_seq=2 ttl=57 time=1.18 ms
    64 bytes from 208.100.39.54: icmp_seq=3 ttl=57 time=1.10 ms
    64 bytes from 208.100.39.54: icmp_seq=4 ttl=57 time=1.11 ms
    ^C
    --- chicago3.voip.ms ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3261ms
    rtt min/avg/max/mdev = 1.107/1.132/1.180/0.044 ms
    [[email protected] ~]# ping chicago4.voip.ms
    PING chicago4.voip.ms (208.100.39.55) 56(84) bytes of data.
    64 bytes from 208.100.39.55: icmp_seq=1 ttl=57 time=1.13 ms
    64 bytes from 208.100.39.55: icmp_seq=2 ttl=57 time=1.15 ms
    64 bytes from 208.100.39.55: icmp_seq=3 ttl=57 time=1.14 ms
    64 bytes from 208.100.39.55: icmp_seq=4 ttl=57 time=1.25 ms
    64 bytes from 208.100.39.55: icmp_seq=5 ttl=57 time=1.21 ms
    ^C
    --- chicago4.voip.ms ping statistics ---
    5 packets transmitted, 5 received, 0% packet loss, time 4071ms
    rtt min/avg/max/mdev = 1.132/1.180/1.252/0.044 ms
    

    response times to offices
    these response times are typical whether or not there are problems being reported.

    [[email protected] ~]# ping suba.domain.com
    PING suba.domain.com (12.xxx.xxx.xxx) 56(84) bytes of data.
    64 bytes from 12.xxx.xxx.xxx: icmp_seq=1 ttl=53 time=8.83 ms
    64 bytes from 12.xxx.xxx.xxx: icmp_seq=2 ttl=53 time=8.90 ms
    64 bytes from 12.xxx.xxx.xxx: icmp_seq=3 ttl=53 time=8.73 ms
    ^C
    --- suba.domain.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2951ms
    rtt min/avg/max/mdev = 8.734/8.823/8.903/0.128 ms
    [[email protected] ~]# ping subb.domain.com
    PING subb.domain.com (71.xxx.xxx.xxx) 56(84) bytes of data.
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=1 ttl=50 time=35.8 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=2 ttl=50 time=35.6 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=3 ttl=50 time=34.2 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=4 ttl=50 time=34.2 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=5 ttl=50 time=34.9 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=6 ttl=50 time=44.2 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=7 ttl=50 time=43.4 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=8 ttl=50 time=35.1 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=9 ttl=50 time=39.0 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=10 ttl=50 time=33.6 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=11 ttl=50 time=34.6 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=12 ttl=50 time=34.8 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=13 ttl=50 time=35.6 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=14 ttl=50 time=36.1 ms
    64 bytes from 71-xxx-xxx-xxx.static.stls.mo.charter.com (71.xxx.xxx.xxx): icmp_seq=15 ttl=50 time=34.2 ms
    ^C
    --- subb.domain.com ping statistics ---
    15 packets transmitted, 15 received, 0% packet loss, time 14639ms
    rtt min/avg/max/mdev = 33.663/36.405/44.212/3.159 ms
    [[email protected] ~]# ping jeff.domain.com
    ping: unknown host jeff.domain.com
    [[email protected] ~]# ping subc.domain.com
    PING subc.domain.com (216.xxx.xxx.xxx) 56(84) bytes of data.
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=1 ttl=52 time=13.3 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=2 ttl=52 time=13.9 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=3 ttl=52 time=13.1 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=4 ttl=52 time=14.4 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=5 ttl=52 time=15.3 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=6 ttl=52 time=13.8 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=7 ttl=52 time=14.4 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=8 ttl=52 time=15.5 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=9 ttl=52 time=13.8 ms
    ^[[A64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=10 ttl=52 time=14.5 ms
    64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=11 ttl=52 time=15.3 ms
    ^C64 bytes from 216.xxx.xxx.xxx.reverse.socket.net (216.xxx.xxx.xxx): icmp_seq=12 ttl=52 time=16.4 ms
    ^C
    --- subc.domain.com ping statistics ---
    12 packets transmitted, 12 received, 0% packet loss, time 11964ms
    rtt min/avg/max/mdev = 13.199/14.536/16.469/0.951 ms
    [[email protected] ~]# ping subd.domain.com
    PING subd.domain.com (216.xxx.xxx.yyy) 56(84) bytes of data.
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=1 ttl=55 time=12.8 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=2 ttl=55 time=12.4 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=3 ttl=55 time=20.0 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=4 ttl=55 time=12.6 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=5 ttl=55 time=12.7 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=6 ttl=55 time=12.5 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=7 ttl=55 time=12.4 ms
    64 bytes from 216-xxx-xxx-yyy.adams.net (216.xxx.xxx.yyy): icmp_seq=8 ttl=55 time=12.9 ms
    ^C
    --- subd.domain.com ping statistics ---
    8 packets transmitted, 8 received, 0% packet loss, time 7632ms
    rtt min/avg/max/mdev = 12.429/13.570/20.022/2.448 ms
    


  • Noisy Neighbor? Submit a ticket with Vultr 🙂



  • @aaronstuder said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    Noisy Neighbor? Submit a ticket with Vultr 🙂

    I had this exact issue a couple months ago with the NJ data center with VULTR. Noisy neighbor. Opened ticket with Vultr and the neighbor was shut down within 10 minutes.



  • Get it figured out?



  • @wrx7m said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    Get it figured out?

    I am not working on this until Tuesday.







  • @dashrender said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    @scottalanmiller said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    @jaredbusch said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    @wrx7m said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    Get it figured out?

    I am not working on this until Tuesday.

    @jaredbusch said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    @wrx7m said in Intermittent call quality issues with a FreePBX instance on Vultr in Chicago:

    Get it figured out?

    I am not working on this until Tuesday.

    Because, eclipse.

    LOL - I.E. customer, we don't care about your ability to use the phone system. 😛

    They already knew I was going to be unavailable. I am working on other things now anyway..



  • I have a client on the same setup with Vultr (Chicago) and Voip.ms (chicago2.voip.ms). They were having the same issue on Friday with calls dropping in and out. This morning I am still getting a couple complaints but not as bad as Friday.



  • So I submitted a ticket with Vultr and they moved me to a new server to see if that would make any difference. The call quality is a little better but there is still a little static/pop every couple of seconds. They had me send MTR results to them. I pointed out that there is some packet loss happening on the last hop but they said everything looks good. Below are my results. I removed the first few lines.

    Host Loss % Sent Recv Best Avrg Wrst Last
    be-33491-cr02.350ecermak.il.ibone.comcast.net 0 36 36 12 16 25 17
    be-10563-pe01.350ecermak.il.ibone.comcast.net 0 36 36 11 15 26 12
    50.242.150.34 0 36 36 11 17 69 13
    0.ae12.cr2.ord6.scnet.net 0 36 36 12 17 34 17
    ae2.ar9.ord6.us.scnet.net 0 36 36 12 16 32 13
    ethernetxe-0-0-0:0-er1-q8.chi2.choopa.net 0 36 36 13 18 54 14
    No response from host 100 7 0 0 0 0 0
    No response from host 100 7 0 0 0 0 0
    myinstance.vultr.com 4 32 31 0 13 20 13
    ________________________________________________ ______ ______ ______ ______ ______ ______

    WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider



  • The issue seems to have been resolved. No packet drops on MTR this time around.



  • Opened a ticket with Vultr. We'll see what they suggest.