Office 365 Email Gone After Forced Logoff
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
When shit blows up on our end, we know it.
Which makes pretending it doesn't happen much worse. Although we've been in meetings with MS where they've claimed that they didn't know it and were surprised to find out that certain outages were common.
It's nice to claim that by being big and rich that you are infallible. But we both know that hubris like that is exactly what makes companies bad at IT. The more hubris there is, the worse the service is.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
When shit blows up on our end, we know it.
Which makes pretending it doesn't happen much worse. Although we've been in meetings with MS where they've claimed that they didn't know it and were surprised to find out that certain outages were common.
It's nice to claim that by being big and rich that you are infallible. But we both know that hubris like that is exactly what makes companies bad at IT. The more hubris there is, the worse the service is.
Ugh. If you think it magically purged, let's see what the logs say.
Login as an administrator, go to Azure Active Directory, then audit log. It will show everything that happened. Look for UpdateServicePrincipal entries. Should say who did what and when:
-
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
If you understood how things work with it, you would know that AAD is the function of O365's service stuff. AD Sync pulls your users into AAD. Federation makes your AD the authoritative resource.
Pull the log, see what you find.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
If you understood how things work with it, you would know that AAD is the function of O365's service stuff. AD Sync pulls your users into AAD. Federation makes your AD the authoritative resource.
Pull the log, see what you find.
But none of that is what we are having issues with.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
If you understood how things work with it, you would know that AAD is the function of O365's service stuff. AD Sync pulls your users into AAD. Federation makes your AD the authoritative resource.
Pull the log, see what you find.
But none of that is what we are having issues with.
You are saying that all your Exchange accounts have popped off your user accounts.
AAD will tell you what you need to know.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
If you understood how things work with it, you would know that AAD is the function of O365's service stuff. AD Sync pulls your users into AAD. Federation makes your AD the authoritative resource.
Pull the log, see what you find.
But none of that is what we are having issues with.
You are saying that all your Exchange accounts have popped off your user accounts.
AAD will tell you what you need to know.
So you are saying that Azure can tell us the billing and account information that even MS couldn't find for three hours? If MS has these resources, why didn't MS use them to determine what was wrong?
-
Someone needs to train the MS employees. Hours and hours today, and MS couldn't find what was wrong for the longest time. Would be nice if MS staff knew that there were ways to just look this up. It took a long time just to find a department that could work on the issue.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector Yes, but most issues are not related to Azure. We get no logs of account issues.
If you understood how things work with it, you would know that AAD is the function of O365's service stuff. AD Sync pulls your users into AAD. Federation makes your AD the authoritative resource.
Pull the log, see what you find.
But none of that is what we are having issues with.
You are saying that all your Exchange accounts have popped off your user accounts.
AAD will tell you what you need to know.
So you are saying that Azure can tell us the billing and account information that even MS couldn't find for three hours? If MS has these resources, why didn't MS use them to determine what was wrong?
Gotta know how to read between the lines. There is deeper info on the MS side of things obviously, but my guess? You don't have a very high support contract level so its just SevC at the moment.
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
How so? Did you find something that did an UpdateServicePrincipal that wasn't executed by you?
Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
How so? Did you find something that did an UpdateServicePrincipal that wasn't executed by you?
As is often the case, the issue was account related on the MS side. Nothing to do with the users on our end. Nothing we had the ability to control. Certainly not executed by us, not even accessible to us.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?
Yup, and that's how it manifested. And quickly it popped off all users. But then turned out to be only partially off, it went from the portal, but the system returned. Turned out to be issues with the partner team and how they applied some account details. But no one at MS could find that, for hours. And none of it was visible to us.
-
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry, because something had to tell your domain that it no longer has Exchange. That feeds back into the portal.office.com site, removing the Outlook link. You can't nuke your service without it.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
Yeah, somewhere on the backend out of our reach or visibility.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, what do you see in AAD?
No activity in the logs from this morning until after the issue had started. And all the first logs are then clearly us trying to figure out what was wrong. So whatever they did, wasn't recorded in there.
-
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
-
Not to be obtuse but why do Zimbra instead of Zoho or another?
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
I’d take Zoho docs over LibreOffice any day.
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.
