Azure Anecdote
-
SAM's recent Azure meltdown brought to mind a somewhat recent event we had with Azure. More specifically, with Visual Studio Online.
We're a small software development company of about 50ish employees at any given time. One of our internal source control platforms is Team Foundation Server. We tend to do a LOT of collaboration work with outside consultants and contractors. Since our TFS is inaccessible from the "Outside", Needless to say, this turns into a headache for me because I have to worry about Acceptable Use stuff, troubleshooting VPN issues, account and access issues, etc. with this "Outside" people. Visual Studio Online (essentially "TFS in the cloud") solves a lot of those issues for me. It's still not as full featured as internal TFS, but it gets the job done admirably most of the time. This leads me to my story...
We are a Silver MS partner. One of the benefits of this is having some free Azure credits. The account we used to sign up for these credits was a "Microsoft account". Those of you who are familiar with this know that there are two kinds of accounts in Microsoft's world - the aforementioned "Microsoft account", and the "Organizational or Work account" (this is essentially your Azure AD account synced from on-prem.) Since we adopted VS Online before the service was "hooked into" Organizational Account login, our account was tied to Microsoft account. At the time, there was no way to mix and match these (there is a way now... sort of.. that's another story for another time lol).
Anyway, I called MS subscription support (as per some conversion documentation on TechNet) to get our account "changed over" to an Organizational account. The woman I spoke with said that it was all taken care of and indeed everything seemed to be fine. Until two weeks later...
Help Desk ticket came in saying that no one can log into VS Online. Sure enough, I verified that we no longer had VS Online in our Azure Subscription (what the HOLY f&$&^???). Where was it?!? Did all my VS Projects just vanish? So, I called support and after almost TWO whole days, we found out that the conversion was never fully done and that they set the old account to expire (which it did, thus the inability to login and the missing VS instance in Azure).
LUCKILY, nothing was actually gone. Once we re-added the instance to the proper subscription, the projects were all back. I still had to re-assign VS subscriptions to each user (what a pain in the ass that was!) in order to get them back up and running again, but everything was back to normal.
So yeah, that was my Azure Anecdote. Sorry for the lack of brevity lol.
-
We've recently made a six figure investment in Microsoft Dynamics NAV and Microsoft and their resellers have been massively promoting using Azure for it. I'm so glad now that I didn't listen to them and went with on-premise.
-
Both this issue and SAM's issue are not outages of the Azure service. They are both account related outages.
Obviously an inaccessible is still an outage from the point of view that services are not available, but it is a slightly different issue compared to Azure going down.
-
While I completely understand Scott's and AVI's unhappiness, I'm not sure they are real reasons to avoid using Azure. One thing it definitely tells you is, if you have any type of account related problem... you're probably best off opening a brand new account and migrating everything over to that account and closing down the questionable accounts. This seems to be the only option that ensures you don't have to worry that someone at MS is actually following through on an account cleanup problem.
-
Downtime is downtime. It really doesn't matter to me (I'm not that technical!) what the cause was. It's either up and available, or it's down and unavailable.
-
@Carnival-Boy said:
Downtime is downtime. It really doesn't matter to me (I'm not that technical!) what the cause was. It's either up and available, or it's down and unavailable.
True, but in relation to how MS reports Azure's uptime, they would not report either of these as a negative in anyway toward that reported number. So if you weren't here reading those posts, you'd have no idea these outages even happened.
It's possible the same thing has happened on Rackspace, it's just posted in places none of us read.
-
Yeah. That's why anecdotal evidence is important and why hanging out on forums is worth it. And why you should take SLAs with a large pinch of salt.
-
@Carnival-Boy said:
And why you should take SLAs with a large pinch of salt.
I never considered that in the past, definitely something I've learned by being here.
-
@Carnival-Boy said:
Downtime is downtime. It really doesn't matter to me (I'm not that technical!) what the cause was. It's either up and available, or it's down and unavailable.
Downtime is downtime form the customer point of view yes. But from the point of view of what the problem is, that is very specific. Azure was never down. No servers or datacenter were offline.
@AVI-NetworkGuy lost access to data because accounts were improperly merged by Microsoft. He was unable to access. Azure was not down. All still a Microsoft f[moderated] up, though.
@scottalanmiller posted an almost click bait worthy title claiming that Azure was down. Azure was never down and there was no outage. He lost access to services because Microsoft closed or deleted the account improperly. This was the access system doing exactly what it was supposed to do. It took things offline because the account was no longer active. The account being no longer active was the wrong part. This is still an outage, but it is not "Azure is down" or even close to that. I have not seen a follow up breakdown of what happened, but one can assume that this was again a f[moderated] up on the part of the Microsoft handling of accounts.
-
Point taken.
-
@JaredBusch It was never implied in my story that the Azure service itself was down (not sure if that's what you're trying to say here). We rely on Azure everyday for a lot of our services and have little to no problems (aside from what I wrote about above which, as you said, was an account-related issue that looks like an "outage" to my users). There are QUITE a few very real outage\performance issues with Visual Studio Online (just look at their Service Blog for evidence of that - http://blogs.msdn.com/b/vsoservice/)
Anyway, the whole point of my story was to show that things can go wrong with the service and that Microsoft can be a bit painful to work with to resolve these issues. That's all
-
I think it was more fo a dig at @scottalanmiller 's choice of thread title, but anyway, these anecdotes are starting to make me something of a cloud sceptic (is that a term?). On the one hand, @scottalanmiller writes "by and large the vast majority of email systems should be hosted", but other the hand he writes "Same issues caused an O365 outage two weeks ago and email data loss. I still am missing most of my email from the last two years."
Missing e-mail is catastrophic for most people, including me. People focus on the technical details of hosting, which are clearly massively reliable, and don't focus on the account details (ie Microsoft refusing you access to their perfectly working systems because of a clerical error). Clearly, I can't make my on-premise systems as reliable as an IT company can make its hosted system, but I can make my clerical systems more reliable. So you have to factor both things in when evaluating the risks of hosting versus on-premise, as we have seen.
-
@JaredBusch said:
Both this issue and SAM's issue are not outages of the Azure service. They are both account related outages.
Obviously an inaccessible is still an outage from the point of view that services are not available, but it is a slightly different issue compared to Azure going down.
Correct, it's a full outage, but not "Azure down" but Azure services were actually down at the account level. Their technical engineers were unable to get it up. So while the physical infrastructure wasn't down across the board, it was actually down for us and at a purely technical level even Microsoft could not get it to come up. So it's more than an account problem, it's an account-related problem that triggered a full technical outage.
-
@Carnival-Boy said:
Yeah. That's why anecdotal evidence is important and why hanging out on forums is worth it. And why you should take SLAs with a large pinch of salt.
The SLA was met, in this case, by them giving us credits for the outage. So while they were not up as much as they were supposed to be, they did not shirk their SLA duties in any way. Not like we had to ask for the credits or anything.
-
@JaredBusch said:
@scottalanmiller posted an almost click bait worthy title claiming that Azure was down. Azure was never down and there was no outage. He lost access to services because Microsoft closed or deleted the account improperly.
That's still Azure down. To us and our customers, Azure was completely down. All servers offline. It's an outage same as any other. It's localized in its impact to an account based "region" rather than to a datacenter-based region or whatever, but it is an outage just the same. The system was supposed to be up, and it was down and Microsoft was unable to bring it up. Even to Microsoft's own technical team it was an outage that they could not fix (for half a day.)
