Cloudatcost

PSX_Defector

@PSX_Defector said:

Although I'm seeing the same thing right now on TWC, haven't checked AT&T.

On SBC / AT&T here.

There's actually two different routes with them. There's ATTIS, which is usually the DSL pipes and some hi-cap, then there is the U-Verse platform which takes a different route. As far as I can see, it's a BGP route issue since it affects multiple providers.

I see this from TWC:

C:\Users\v436525>tracert jump.ntg.co

Tracing route to jump.ntg.co [168.235.144.189]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms agrer003-ip002001.noa.vmotion.tmrk.eu [172.16.2.
1]
2 15 ms 26 ms 17 ms cpe-76-186-176-1.tx.res.rr.com [76.186.176.1]
3 10 ms 11 ms 10 ms tge7-2.allntx3901h.texas.rr.com [24.164.210.241]

4 14 ms 15 ms 15 ms tge0-8-0-7.plantxmp01r.texas.rr.com [24.175.37.2
12]
5 13 ms 15 ms 15 ms agg27.crtntxjt01r.texas.rr.com [24.175.36.177]
6 * * * Request timed out.

My route from AT&T don't want to come through on pfSense right now. I can certainly generate it once I get off online and just hard drop the AT&T line. Still technically have a few minutes to go at the big red V. Although I am super drunk right now.

People are now reporting Cell Phone outages..

http://www.akamai.com/html/technology/dataviz1.html

Interesting. They are reporting 53% above average attack traffic in North America today.. No idea of the relevance.

scottalanmiller

@thecreativeone91 said:

People are now reporting Cell Phone outages..

Television too.

Well hurricane Electric routing tables have been completely flushed of all routes there. I tried a trace route from one of their routers and it just fails with no route on the first hop.

AmanBhogal

Hey Everybody!
Late night update for you all. It's almost 9pm and we are all scrambling to make sure everything gets resolved ASAP.

Rogers is telling us that we should be up and running by midnight EST tonight.

As you can all understand, this was a pretty heavy hit for us today and any time something like this happens, its always unfortunate. No company is too big or too small to avoid these failures, and seeing Rogers experience massive network issues is a prime example of that.

For me personally, this could not have come at a worse time. I really enjoy interacting with each and every one of you, reading your posts about cloudatcost, seeing comments about how you are planning on using your servers, then all of a sudden... BOOM!

Regardless, I am still here to answer any questions you may have and I will make sure that once the dust has settled on this, we do something awesome for ML users.

Comparing Cloudatcost to many of the names you knew before you heard of us, we are small... but we are determined, and we WILL learn from this, and we WILL find a way to make it so that next time this happens, we are able to get everybody back up sooner and quicker than today.

mlnews

AmanBhogal

Hey Everybody,
I am happy to announce that the issue appears to be resolved.

Yay, Finally.. Only 13+ Hrs later.

scottalanmiller

I'm back into the servers!

StrongBad

Looks like I missed a lot of excitement today.