Office 365 Email Gone After Forced Logoff
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
How so? Did you find something that did an UpdateServicePrincipal that wasn't executed by you?
Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.
Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.
How so? Did you find something that did an UpdateServicePrincipal that wasn't executed by you?
As is often the case, the issue was account related on the MS side. Nothing to do with the users on our end. Nothing we had the ability to control. Certainly not executed by us, not even accessible to us.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?
Yup, and that's how it manifested. And quickly it popped off all users. But then turned out to be only partially off, it went from the portal, but the system returned. Turned out to be issues with the partner team and how they applied some account details. But no one at MS could find that, for hours. And none of it was visible to us.
-
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry, because something had to tell your domain that it no longer has Exchange. That feeds back into the portal.office.com site, removing the Outlook link. You can't nuke your service without it.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
Yeah, somewhere on the backend out of our reach or visibility.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, what do you see in AAD?
No activity in the logs from this morning until after the issue had started. And all the first logs are then clearly us trying to figure out what was wrong. So whatever they did, wasn't recorded in there.
-
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
-
Not to be obtuse but why do Zimbra instead of Zoho or another?
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
I’d take Zoho docs over LibreOffice any day.
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.
Oh yeah. I forgot you had the big Scale cluster. I was thinking you were running on a VPS.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?
Oh oh, it's magic!
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?
Oh oh, it's magic!
We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...
So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.
Hey at least you didn't lose anything.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...
So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.
Thinking about this on the drive home, since you have a partner account, it might be in another area.
Instead of going round and round with your non-stop conspiracy shit, I'm giving up. Given you didn't even know about AAD, I'm going with you don't know what you are doing. You don't have the right info in front of you and I'm not about to do your job.
You keep thinking MS is out to get ya.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Instead of going round and round with your non-stop conspiracy shit, I'm giving up.
To what are you referring? You asked me to provide logs. You said that all actions must be in the logs. You got the logs that showed nothing. You are calling "showing you the logs that you requested" a conspiracy?
I don't follow. Please explain where conspiracy comes into the picture. I did what you asked. You alone are freaking out that you think that there is some coordinated plan to make things not work and make MS look bad or something. I just told you how it worked. That you were not familiar with O365 and Azure logs is not a conspiracy.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
You keep thinking MS is out to get ya.
Um, only you said that. I said they had a problem. You seem to feel that anyone having any service problem from MS must mean that there is a conspiracy?
Those are your logs that are empty. My contention is that Microsoft doesn't know how to support their platforms. It's not a conspiracy, I just think MS isn't very competent. And being 100% convinced that the logs can't be wrong, finding them empty, and then freaking out that it must be a coordinated effort by customers to modify the logs kind of proves my point.
Just because someone isn't a masterful engineer does not a conspiracy make.
-
@aaronstuder said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller Why are you still using Office 365 if you have so many issues with it?
We managed to get migrated off of it faster than MS was able to fix it
I'm super thiankful for this outage as the timing was perfect to make the decision to drop O365 immediately with all of the decision makers in the right place. And MS' response here really solidifies the decision. Not how you want a vendor responding. Everyone makes mistakes, but MS didn't handle the embarrassment well at all.So we are over on Zimbra now and things are great!
