As one of my co-workers quipped "They're not getting any malware now!"

So we did not have social media for a while. The world didn't end is okay to take a break from seen what other people are doing with their lives.

@scotth said in Outage - Armstrong in N-Western PA:

Could easily be.
I'm not sure (maybe Windstream -- ugh) is the next hop up from one of our sites in NE Ohio and they were down all day Monday.
Some odd behavior with our cell service lately too. We have dead spots where we have never had issues before. Verizon, in this case.

The storms that came through earlier this week wrecked havock on infrastructure in NE Ohio. We still have roads closed and people we know are not due to get power restored till this weekend.

At work I've cleared a record number of tickets this week because we had so many outages.

@skyetel said in Did you notice the Skyetel outage today?:

I put this on another forum, but I figured you guys would like to read this too:

For those interested, Our network operates in 4 AWS regions that we keep siloed from one another (meaning no region knows about the existence of another one). When the connectivity issues started, we disabled the network connectivity for all Skyetel assets in the two impacted AWS regions which caused our network to fully failover. (Because the impacted regions had partial connectivity, our network did not fully fail over and tried to limp along with all 4. This is by design; we don't want to automate disabling network routers of our network for obvious reasons... so an engineer needed to click the buttons).

The impact of this was some calls failed to establish, but if they did establish, they would work normally. This is because we are not in the audiopath of the calls. Once the distressed regions were fully down, our network could fully fail over and 100% of all calls completed normally.

The total impact time was 19 minutes, and we estimate about 7% of our calls failed to establish during that period. Sorry for the inconvenience 🙂

Did I mention @Skyetel is awesome lately? Being fixed almost before the customers noticed was insanely awesome.

@scottalanmiller I agree and upon reflection, using Vultr DDoS wouldn't work if they were attacked anyhow. Certainly it would be higher up the food chain. I have backups of everything and would have it up very quick somewhere else. I've actually been working on my Kazoo setup and can use it as failover.

We use Google services here. We had a quirk around 6am-7am on Monday. No end-user issues reported since that 1 hour. However all of our facilities are within TX.

@VoIP_n00b said in Significant Google Outage this AM:

@Dashrender said in Significant Google Outage this AM:

@gjacobse said in Significant Google Outage this AM:

Kids school had sent out notice of issue and shortly after that it should be resolved.

Sad- one service “took out” school systems across the US.

This is the centralized world we're moving to.

Decentralized is really the system we want.

Why? Decentralized means more, smaller outages but doesn't solve the fundamental issues. In fact, it would make them worse. Decentralized outages don't make headlines, but they cause more downtime to the end users. More downtime, with far less ability to point to and resolve. They require vastly more resources so the ability to repair is much lower.

Still down.

https://status.vultr.com

@Emad-R said in Whats up with CloudFlare CDN:

@scottalanmiller

This site reports 2 as well:
https://bgr.com/2019/07/02/internet-outage-2019-cloudflare-network-issues/

They post two yes but not all due to Cloudflare,

June 24, 2019 - Route Leak Impacting Cloudflare (Other ISPs involved)
https://www.cloudflarestatus.com/incidents/46z55mdhg0t5

July 2nd, 2019 -HTTP 502 errors
https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

It's been a number of years now, but https://viviotech.net/windows-vps-hosting.html did a good job when I was supporting companies on their Windows platform.

@JaredBusch said in Major outage at VoIP.ms today:

@scottalanmiller said in Major outage at VoIP.ms today:

Wait, I think we are talking on a voip.ms line right now, weird.

The problem is inbound termination from upstream of them. Everything else works.

Also not all inbound providers are affected.

Okay, that makes sense then.

If ever there was an RGE*, this would be it.
 
 
 
 
*Resume Generating Event

@BRRABill said in Crashplan Pro Issue:

@JaredBusch said

OMFG how can you even say that you are not sure a restore is related? Seriously?

Do you know files can't be restored? Maybe there is a way to do it outside the normal ways. Who knows? I have not officially asked about that.

/blink

Umm, if the backup repository is unavailable for backup, how in the hell would it be available for a restore?

@scottalanmiller said in O365 and Outlook - Trying to Connect:

@DustinB3403 said in O365 and Outlook - Trying to Connect:

Day 4 and and Autodiscover is still broken. Microsoft should be calling me any minute.

Did they?

Ha.. yes they did and there were additional configuration changes made so that users are authentication against out on premise server, but the cached records that O365 collects.

And there is a new issue that I've been dealing with for a few days (4) were we apparently can't sync users / password / dl changes etc.

Even though the sync says everything is functional.

Yeah I just tried and am at able to sign into my steam account.

Testing a few games now.

@Dashrender said in CenturyLink Level 3 November 22 2016 Outage:

@scottalanmiller said in CenturyLink Level 3 November 22 2016 Outage:

All three, Qwest, CenturyLink and Level 3 are the same company. Qwest is the "parent" but used CL as their name.

OK there we go... I didn't know that QWest, CenturyLink and Level 3 are the same company.

One big, crappy company that is always offline, apparently.

So our service appears to be restored.

Also WebEx is now directing calls to "Due to high call volume we can't answer you call, please try again later." and than they hang up.

So there is that...