Cloudatcost currently offline


  • Vendor

    Cloudatcost is currently offline due to an upstream provider.
    I will keep everybody posted on network status as soon as I receive updates.
    We are currently working on a resolution.


  • Service Provider

    Thanks for the direct information!

    Also: https://twitter.com/cloudatcost


  • Vendor

    Hey Everybody,
    the issue has been isolated to one of our data providers. We are currently working on resolving the issue and moving over to backup connections.



  • Sorry to hear that! Keep us posted!



  • This is bad, really bad!



  • @IRJ said:

    This is bad, really bad!

    Yeah you can check outages here: https://canadianoutages.com/status/rogers/toronto This is mostly on Rodgers and you'd think they'd have an auto fail over to another connection or at least an easy process in place to switch over.

    I really want another free Dig Dog 1/2 instance out of this.. Just saying... Unacceptable



  • My VPS is back up up but the management panel is still down. Total downtime was 5 hours. That is a crazy amount of down time for an unplanned outage. Although I understand it is mostly on Rodgers.



  • Most likely a call before you dig error but crazy that they have no other fiber lines into the place



  • @MattSpeller said:

    Most likely a call before you dig error but crazy that they have no other fiber lines into the place

    Wouldn't doubt it, have had that happen here a couple times too. But not having an automated failover for a cloud service seems a bit far fetched.


  • Service Provider

    That's a huge coincidence that CloudatCost has ISP issues on the same morning that Rackspace had to do rolling restarts.



  • I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.



  • @Danp said:

    I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.

    I can't either


  • Service Provider

    @thecreativeone91 said:

    Also if they are owned by Fibernetics why is the WHO IS info not fibernetics?

    WHO IS doesn't mean anything. My name is personally on all kinds of companies because lots of companies proxy through NTG and NTG uses an account from the 1990s that had my name everywhere because I was the technical contact. So even companies that have no more relationship with NTG other than using them for some aspect of technical support often look like they are set up by me just because of how registrar accounts are handled.


  • Vendor

    Hey Everybody!

    This was definitely unplanned and unacceptable.
    There was a fiber outage with Rogers which affected users all across Ontario. it is good to know that we aren't the only ones affected by this outage.

    Currently looking into getting more answers


  • Service Provider

    @Danp said:

    I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.

    It's an ISP outage, so that makes sense. I don't believe that there is any concern that the boxes themselves have gone down.


  • Service Provider

    @coliver said:

    @MattSpeller said:

    Most likely a call before you dig error but crazy that they have no other fiber lines into the place

    Wouldn't doubt it, have had that happen here a couple times too. But not having an automated failover for a cloud service seems a bit far fetched.

    We had a colo in the mid-2000s that had fiber failover. But due to a massive backhoe incident in another city, both of their providers were cut at once because they had not disclosed that while they were separate ISPs, in separate cities, that they shared conduit!



  • @scottalanmiller said:

    @Danp said:

    I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.

    It's an ISP outage, so that makes sense. I don't believe that there is any concern that the boxes themselves have gone down.

    Yes, but why are they blaming it on rodgers now when, before when promoting it they were saying they were their own ISP with Fibernetics had their own very robust backbone.



  • @AmanBhogal said:

    Hey Everybody!

    This was definitely unplanned and unacceptable.
    There was a fiber outage with Rogers which affected users all across Ontario. it is good to know that we aren't the only ones affected by this outage.

    Currently looking into getting more answers

    Good to hear. When/if you get more answers an incident report would be much appreciated. Only if you are allowed to.


  • Service Provider

    @thecreativeone91 said:

    Yes, but why are they blaming it on rodgers now when, before when promoting it they were saying they were their own ISP with Fibernetics had their own very robust backbone.

    But even as an ISP, they still need peering to get Internet access. Even ISPs go down.



  • @thecreativeone91 said:

    @scottalanmiller said:

    @Danp said:

    I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.

    It's an ISP outage, so that makes sense. I don't believe that there is any concern that the boxes themselves have gone down.

    Yes, but why are they blaming it on rodgers now when, before when promoting it they were saying they were their own ISP with Fibernetics had their own very robust backbone.

    Doesn't Canada only have one or two backbone providers? Everyone else just purchases bandwidth from them? That is probably wrong just thought I remembered hearing that.



  • @scottalanmiller said:

    @thecreativeone91 said:

    Yes, but why are they blaming it on rodgers now when, before when promoting it they were saying they were their own ISP with Fibernetics had their own very robust backbone.

    But even as an ISP, they still need peering to get Internet access. Even ISPs go down.

    Yes, but if they only have one peering connection something is wrong. Any Fiber provider I've used has at least 4-5 peering connections.


  • Service Provider

    @coliver said:

    @thecreativeone91 said:

    @scottalanmiller said:

    @Danp said:

    I can now log into my panel, which shows that both of my boxes are up and running. However, I still can't connect to them.

    It's an ISP outage, so that makes sense. I don't believe that there is any concern that the boxes themselves have gone down.

    Yes, but why are they blaming it on rodgers now when, before when promoting it they were saying they were their own ISP with Fibernetics had their own very robust backbone.

    Doesn't Canada only have one or two backbone providers? Everyone else just purchases bandwidth from them? That is probably wrong just thought I remembered hearing that.

    Two, yes. Rogers and Bell Canada. NTG's colo sits in 151 Front Street, the hotel and homebase of both. So our servers go straight to both of them. It's pretty awesome.


  • Service Provider

    @thecreativeone91 said:

    Yes, but if they only have one peering connection something is wrong. Any Fiber provider I've used has at least 4-5 peering connections.

    The best that they could reasonably get is two. Canada is much more susceptible to big outages than we are in the US in that way.



  • @AmanBhogal said:

    Hey Everybody!

    This was definitely unplanned and unacceptable.
    There was a fiber outage with Rogers which affected users all across Ontario. it is good to know that we aren't the only ones affected by this outage.

    Currently looking into getting more answers

    Thank you for the update! I keep a few of these around, feel free to borrow one on your hunt for answers 😃

    -----E
    -----E
    -----E



  • @scottalanmiller said:

    @thecreativeone91 said:

    Yes, but if they only have one peering connection something is wrong. Any Fiber provider I've used has at least 4-5 peering connections.

    The best that they could reasonably get is two. Canada is much more susceptible to big outages than we are in the US in that way.

    Good to know



  • @MattSpeller said:

    @AmanBhogal said:

    Hey Everybody!

    This was definitely unplanned and unacceptable.
    There was a fiber outage with Rogers which affected users all across Ontario. it is good to know that we aren't the only ones affected by this outage.

    Currently looking into getting more answers

    Thank you for the update! I keep a few of these around, feel free to borrow one on your hunt for answers 😃

    -----E
    -----E
    -----E

    It took me a bit... are those pitchforks?



  • @scottalanmiller said:

    The best that they could reasonably get is two. Canada is much more susceptible to big outages than we are in the US in that way.

    Unlike the USA where you have lots of crisscrossing up and down runs, Canada is (mostly) just east to west, within 100km of the border with the USA. Two backbones is not too shabby considering the length of the run and the number of customers served. Wish it was different but c'est la vie.


  • Service Provider

    @MattSpeller said:

    Unlike the USA where you have lots of crisscrossing up and down runs, Canada is (mostly) just east to west, within 100km of the border with the USA. Two backbones is not too shabby considering the length of the run and the number of customers served. Wish it was different but c'est la vie.

    Yes, very important for Americans to remember that we are talking about a country that is smaller (in population) that California. It is between Texas and California in terms of population.



  • @scottalanmiller Yup, just slightly less (~36mil) than all of California (~39mil)


  • Service Provider

    @MattSpeller said:

    @scottalanmiller Yup, just slightly less (~36mil) than all of California (~39mil)

    It's like a yard to California's meter.