Office 365 Email Gone After Forced Logoff



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @psx_defector said in Office 365 Email Gone After Forced Logoff:

    Azure, by extension O365, unless you pay for support, is self service. Lots of things you need to do for yourself. But on the other side, there is TONS of info you can find on your own. You just need someone to show you how.

    Well, that's only so useful when the end result is "stuff on the MS side." So knowing that for sure from AAD, or not, same thing. Gotta call in and wait for support to get their stuff together. Going into AAD wouldn't have gotten us any further since the issues weren't on our end.

    How so? Did you find something that did an UpdateServicePrincipal that wasn't executed by you?

    As is often the case, the issue was account related on the MS side. Nothing to do with the users on our end. Nothing we had the ability to control. Certainly not executed by us, not even accessible to us.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?

    Yup, and that's how it manifested. And quickly it popped off all users. But then turned out to be only partially off, it went from the portal, but the system returned. Turned out to be issues with the partner team and how they applied some account details. But no one at MS could find that, for hours. And none of it was visible to us.



  • So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?

    There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.

    Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry, because something had to tell your domain that it no longer has Exchange. That feeds back into the portal.office.com site, removing the Outlook link. You can't nuke your service without it.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?

    Yeah, somewhere on the backend out of our reach or visibility.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    Again, what do you see in AAD?

    No activity in the logs from this morning until after the issue had started. And all the first logs are then clearly us trying to figure out what was wrong. So whatever they did, wasn't recorded in there.



  • Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.

    0_1523483085076_Screenshot from 2018-04-11 16-44-24.png

    I posted on here at 9:20am, 12 minutes before the first log entry in AAD.



  • Not to be obtuse but why do Zimbra instead of Zoho or another?



  • @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    Not to be obtuse but why do Zimbra instead of Zoho or another?

    I’d take Zoho docs over LibreOffice any day.



  • @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    Not to be obtuse but why do Zimbra instead of Zoho or another?

    We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    Not to be obtuse but why do Zimbra instead of Zoho or another?

    We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.

    Oh yeah. I forgot you had the big Scale cluster. I was thinking you were running on a VPS.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.

    0_1523483085076_Screenshot from 2018-04-11 16-44-24.png

    I posted on here at 9:20am, 12 minutes before the first log entry in AAD.

    Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?

    Oh oh, it's magic!



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.

    0_1523483085076_Screenshot from 2018-04-11 16-44-24.png

    I posted on here at 9:20am, 12 minutes before the first log entry in AAD.

    Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?

    Oh oh, it's magic!

    We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.

    Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...

    So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.

    Hey at least you didn't lose anything.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @psx_defector said in Office 365 Email Gone After Forced Logoff:

    There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.

    Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...

    So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.

    Thinking about this on the drive home, since you have a partner account, it might be in another area.

    Instead of going round and round with your non-stop conspiracy shit, I'm giving up. Given you didn't even know about AAD, I'm going with you don't know what you are doing. You don't have the right info in front of you and I'm not about to do your job.

    You keep thinking MS is out to get ya.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    Instead of going round and round with your non-stop conspiracy shit, I'm giving up.

    To what are you referring? You asked me to provide logs. You said that all actions must be in the logs. You got the logs that showed nothing. You are calling "showing you the logs that you requested" a conspiracy?

    I don't follow. Please explain where conspiracy comes into the picture. I did what you asked. You alone are freaking out that you think that there is some coordinated plan to make things not work and make MS look bad or something. I just told you how it worked. That you were not familiar with O365 and Azure logs is not a conspiracy.



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    You keep thinking MS is out to get ya.

    Um, only you said that. I said they had a problem. You seem to feel that anyone having any service problem from MS must mean that there is a conspiracy?

    Those are your logs that are empty. My contention is that Microsoft doesn't know how to support their platforms. It's not a conspiracy, I just think MS isn't very competent. And being 100% convinced that the logs can't be wrong, finding them empty, and then freaking out that it must be a coordinated effort by customers to modify the logs kind of proves my point.

    Just because someone isn't a masterful engineer does not a conspiracy make.



  • @aaronstuder said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller Why are you still using Office 365 if you have so many issues with it?

    We managed to get migrated off of it faster than MS was able to fix it 🙂 I'm super thiankful for this outage as the timing was perfect to make the decision to drop O365 immediately with all of the decision makers in the right place. And MS' response here really solidifies the decision. Not how you want a vendor responding. Everyone makes mistakes, but MS didn't handle the embarrassment well at all.

    So we are over on Zimbra now and things are great!



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    And MS' response here really solidifies the decision

    I'm not going to get into your little spat but this point needs made.

    An employee's personal response is not a vendor response. Pull your head out of your ass.



  • @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.

    You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.



  • @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.

    You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.

    No idea what happened to that middle sentence. I meant to say I doubt MS has agreed for him to represent them let alone even know that someone who works for them has even responded in here.



  • @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.

    You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.

    Except that's just a handy way to have employees who represent you that you can then disavow if they do something that you don't like. And it is really clear that he was here purely to try to defend MS. It's not like it was unrelated to his employer, it's not like it wasn't an emotional outburst on their behalf. It's not like they've stepped in and said "whoa, that's not official." He got involved attempting to represent MS, whether "they" wanted him to or not. And what defines "they" other than being an employee?



  • @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.

    You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.

    No idea what happened to that middle sentence. I meant to say I doubt MS has agreed for him to represent them let alone even know that someone who works for them has even responded in here.

    Well they are aware of the community and have been invited to participate, there is no limits or fees preventing them from doing so, they can openly respond right now. They have an employee in the community who decided to speak out very overtly on their behalf and attempt to shame their customers. In this case "they" have an employee here.

    The problem is, if he does something good, MS gets credit. If he attacks customers, we just say "well MS didn't authorize him". We don't know that they did or didn't, what we do know is that they've not disavowed him yet.

    The vague "they" here is a problem. What makes "someone" at MS more official than the MS employee participating and speaking on their behalf? What defines an official response versus one we are supposed to ignore? This is a very public, very voluntary MS response.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @stacksofplates said in Office 365 Email Gone After Forced Logoff:

    @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    @jaredbusch said in Office 365 Email Gone After Forced Logoff:

    An employee's personal response is not a vendor response. Pull your head out of your ass.

    They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.

    He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."

    Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.

    The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.

    You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.

    No idea what happened to that middle sentence. I meant to say I doubt MS has agreed for him to represent them let alone even know that someone who works for them has even responded in here.

    Well they are aware of the community and have been invited to participate, there is no limits or fees preventing them from doing so, they can openly respond right now. They have an employee in the community who decided to speak out very overtly on their behalf and attempt to shame their customers. In this case "they" have an employee here.

    The problem is, if he does something good, MS gets credit. If he attacks customers, we just say "well MS didn't authorize him". We don't know that they did or didn't, what we do know is that they've not disavowed him yet.

    The vague "they" here is a problem. What makes "someone" at MS more official than the MS employee participating and speaking on their behalf? What defines an official response versus one we are supposed to ignore? This is a very public, very voluntary MS response.

    No one spoke on their behalf, or even implied that they did.



  • OK, let's put this shit to rest.

    I NEVER said who I work for, be it whomever I work for now, my previous employers, etc. etc. etc. I know I let it loose a few times, but that was well after I left their employ, e.g. Big Red V. I never have or will use my internal knowledge of places to my advantage in public forums, because I know lots of people and have lots of internal knowledge of various processes and procedures.

    My knowledge of the Microsoft ecosystem is my own, because I'm damn good at working with it and have lots of information. My personal lab is setup in Azure. I extend my personal environment with O365 vertical services including AAD. I use Sharepoint Online for development and learning purposes. So I don't have to work for anyone to know everything I need to know.

    For all anyone knows, I might work for myself, or work for a vendor, or work for the government. But you will NEVER get an official "I work at blah". So you might have some IRL knowledge, but when I write as PSX_Defector, I speak only for myself. Never question my integrity or imply any official capacity again.



  • @psx_defector so you work for the feds eh. . . nice cover story!



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    OK, let's put this shit to rest.

    I NEVER said who I work for, be it whomever I work for now, my previous employers, etc. etc. etc. I know I let it loose a few times, but that was well after I left their employ, e.g. Big Red V. I never have or will use my internal knowledge of places to my advantage in public forums, because I know lots of people and have lots of internal knowledge of various processes and procedures.

    My knowledge of the Microsoft ecosystem is my own, because I'm damn good at working with it and have lots of information. My personal lab is setup in Azure. I extend my personal environment with O365 vertical services including AAD. I use Sharepoint Online for development and learning purposes. So I don't have to work for anyone to know everything I need to know.

    For all anyone knows, I might work for myself, or work for a vendor, or work for the government. But you will NEVER get an official "I work at blah". So you might have some IRL knowledge, but when I write as PSX_Defector, I speak only for myself. Never question my integrity or imply any official capacity again.

    In which case, don't make claims on behalf of an organization. You were pretty clear that you knew things only MS could know like how things "always" work and that there were no outages and all claims of such constitute some conspiracy. Don't act as the voice of a vendor and make claims as such. You were awfully certain you knew what HAD to be the case. You were quick to try to discredit without knowing the situation nor understanding how things work on the vendor side. And jumping to "conspiracy" as the only reason that you might have been wrong about what is logged or what actions is logged... bottom line, claiming that your posts are person and not that of a vendor only after it makes them look bad rather than before, and only when it would have defended them had they been correct but embarrassing when they were wrong, doesn't fly.

    You have to choose, are you posting personally or defending a vendor with insider knowledge before you post, not after. And if posting personally, you need to do so in a personal manner. Not as a reaction to the vendor looking bad.



  • @scottalanmiller said in Office 365 Email Gone After Forced Logoff:

    The vague "they" here is a problem. What makes "someone" at MS more official than the MS employee participating and speaking on their behalf?

    What would make someone an official spokesperson on the behalf of any company is an introduction such as:

    "Hi I'm Bob from Microsoft, We were introduced to MangoLassi.it and thought it would be a great place to discuss projects, software etc

    . . . . . . . . . . . . . . . . . . . . . . . . "

    If such a thing has never occurred, any discussion of a someone's place of employment is purely banter and anything ever discussed by said OP is to be taken as any other random person making a post about the color of the grass. . .



  • @psx_defector said in Office 365 Email Gone After Forced Logoff:

    There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.

    Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry, because something had to tell your domain that it no longer has Exchange. That feeds back into the portal.office.com site, removing the Outlook link. You can't nuke your service without it.

    This post. This information is insider employee only stuff. Only someone inside MS could know what is "always" logged by MS' system. Of course, that information turned out to be incorrect, but this post is not stuff that outsides, even partners, would have access to know. Because it is all "backend logging" information that is option on MS' side. MS decides what is recorded, and MS decides what is exposed, and MS decides what can and can't be done without showing up there.