Did you notice the Skyetel outage today?
-
I barely did. @Skyetel is awesome.
I started getting alerts from 2 systems about calls failing to dial @ 09:38 CST.
At 09:51 CST, the Skyetel trunks dropped and came back up.
No call issues afterwards. I didn't even have time to diagnose and figure out what was happening.
-
Same.
-
Yes, was very nice on two fronts. First off - they had a plan of attack for such an incident occurring, and their plan worked out just as written up.
Secondly, and almost as important was their communication to their client base letting them know what was happening with each step along the way.
Very happy and impressed with them.
-
yes we definitely noticed.
users were calling me mins before the notices from Skyetel were being sent out.
-
@krzykat said in Did you notice the Skyetel outage today?:
Yes, was very nice on two fronts. First off - they had a plan of attack for such an incident occurring, and their plan worked out just as written up.
Secondly, and almost as important was their communication to their client base letting them know what was happening with each step along the way.
Very happy and impressed with them.
Agreed - the constant stream of updates keep my management as happy as they could be considering being in an outage.
-
@dashrender said in Did you notice the Skyetel outage today?:
yes we definitely noticed.
users were calling me mins before the notices from Skyetel were being sent out.
Yeah, Skyetel had the status message up and a fix SO fast BUT... with as many customers as we have, we were inundated with tickets and calls about the outage (because calls between customers and us were still working) before the route around was able to take effect. Luckily we spend most of the time just telling people to "check again, we think it is already fixed" rather than "we have no idea how long this will take."
-
I put this on another forum, but I figured you guys would like to read this too:
For those interested, Our network operates in 4 AWS regions that we keep siloed from one another (meaning no region knows about the existence of another one). When the connectivity issues started, we disabled the network connectivity for all Skyetel assets in the two impacted AWS regions which caused our network to fully failover. (Because the impacted regions had partial connectivity, our network did not fully fail over and tried to limp along with all 4. This is by design; we don't want to automate disabling network routers of our network for obvious reasons... so an engineer needed to click the buttons).
The impact of this was some calls failed to establish, but if they did establish, they would work normally. This is because we are not in the audiopath of the calls. Once the distressed regions were fully down, our network could fully fail over and 100% of all calls completed normally.
The total impact time was 19 minutes, and we estimate about 7% of our calls failed to establish during that period. Sorry for the inconvenience
-
@skyetel said in Did you notice the Skyetel outage today?:
I put this on another forum, but I figured you guys would like to read this too:
For those interested, Our network operates in 4 AWS regions that we keep siloed from one another (meaning no region knows about the existence of another one). When the connectivity issues started, we disabled the network connectivity for all Skyetel assets in the two impacted AWS regions which caused our network to fully failover. (Because the impacted regions had partial connectivity, our network did not fully fail over and tried to limp along with all 4. This is by design; we don't want to automate disabling network routers of our network for obvious reasons... so an engineer needed to click the buttons).
The impact of this was some calls failed to establish, but if they did establish, they would work normally. This is because we are not in the audiopath of the calls. Once the distressed regions were fully down, our network could fully fail over and 100% of all calls completed normally.
The total impact time was 19 minutes, and we estimate about 7% of our calls failed to establish during that period. Sorry for the inconvenience
Did I mention @Skyetel is awesome lately? Being fixed almost before the customers noticed was insanely awesome.
