Need advise, restoring domain controller and email server.
-
@Dashrender said in Need advise, restoring domain controller and email server.:
Well this is a pretty bad situation.
Personally I would open up a ticket with Microsoft for $250 and see what their recommendations are.New price is now at $499.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
This morning, I noticed the exchange server have only 5% of free space left, I tried to free up some space by deleting old administrators users inbox, but exchange will not let me saying I don't have the permission to do so. Following https://community.spiceworks.com/topic/386021-cannot-delete-ad-user-insufficent-rights-or-protected-from-accidental-deletion
Strange, because you later say:
I'm in the process of restoring Exchange to the yesterday's 7pm backup now, which will take several more hours before it finish.
Which usually fixes the disk space issue. I'm guessing you are using Veeam or some other VM level backup rather than the proper application level backup which truncates the logs on the Exchange EDB.
On the plus side, assuming your restore works correctly and most of the stuff is in place, you might not need to do much more than that. But knowing the Exchange part of this, I'm guessing you also didn't backup your AD correctly either for an authoritative restore.
You are in for a rough night my friend. First and foremost, if the user objects have not been deleted, don't start now. Exchange ties into the GUID, so not having it will cause your restore to pretty much be useless. If you already deleted them, you are looking at some really messed up stuff. If you have good backups, you might have to straight up restore to a previous point in time completely, that means nuking everything. If you had a block level backup, you might be almost OK then, assuming you don't have too much of a delta between the machines.
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
-
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
I spent 90 minutes on the phone and for just Microsoft so say setup is unsupported.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
I spent 90 minutes on the phone and for just Microsoft so say setup is unsupported.
That's kind of how MS roles. Support is not their forte. Highly recommended if you feel that support is critical, MS is not the place to be depending on.
-
The conclusion was to abandon Site A's DC1. Make Site B's DC2 a primary DC and seize the FSMO since I screwed up by changing permission, screwed up even more by taking out DC1 and restore it from a VM image backup and MS couldn't help.
Then, I talked to my old boss and an idea came to me, "What if I just call MS for help with Exchange and nothing else." So I spent just over 3 hours on the phone with the support guy from India, Neel Kamal Sharma Engineer -Microsoft Enterprise Communication Support. He was VERY knowledgeable about Exchange. Then we found out the Exchange 2010 we have is only a SP2, which is not supported by MS. He upgrade it to SP3, which took over 90 minutes, then he reconnected the lost mailboxes, set the permissions, and Exchange was running fully again. I restored what I can from GFI archiver so minimal lost on emails.
Though this incident, I learned
-
All our VM guest servers are backed thru Barracuda, which takes snap shot then backup the server. Great for an application server. It does NOT work for DC because once you restore the DC, the GUID changes, and AD on that DC will be broken.
-
Our way of backing up Exchange created logs that continue to take up storage space since the backup does not delete the logs.
-
Hosting internal Exchange can be very dangerous thing, since one wrong permission change can wipe out your entire mailboxes.
and many other things.
I still got a few things I need to fix, but at least email is working now.
Then, I will be proposing some new recommendations to management from what I learn.
Thanks for all those who helped. -
-
Great to hear you got an MS tech who was willing to help.
This has been my experience as well. They seemed to bend over backwards to assist in resolving my issues.
Sadly - it seems Scott has not had this experience.
-
@Dashrender said in Need advise, restoring domain controller and email server.:
Great to hear you got an MS tech who was willing to help.
This has been my experience as well. They seemed to bend over backwards to assist in resolving my issues.
Sadly - it seems Scott has not had this experience.
It's like playing whack-a-mole, they do have some good people. The problem is, it's only some, and trying to find a good one is always difficult.
shibboleet
-
On the first call, I was trying to get the Active Directory permission restored from the VM image base backup, then I though I can bring the restored exchange back online. It is just not possible. I tried it, and AD failed immediately with just the restored DC1 being on. It's not the tech's fault for my screw ups.
Second call was to only focusing on Exchange Server, and that ended well.
-
you can definitely put yourself into a situation where they can't/won't help you.. but that's normally because you started doing things you shouldn't and you've actually made the call to support to late. This you only have yourself to blame.
I'm guessing if this type of thing ever happens again, the first thing you'll do is call and open a ticket, either with a support company or Microsoft direct.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
- All our VM guest servers are backed thru Barracuda, which takes snap shot then backup the server. Great for an application server. It does NOT work for DC because once you restore the DC, the GUID changes, and AD on that DC will be broken.
All VM backup solutions work by making snapshots to freeze the disk state that is going to be backed up.
Why your restore changed some GUID is an unrelated issue.
I have backed up and restored a DC, many, many times over the years and never had this kind of problem.
Hell, I have done a backup and restore of a SBS server with zero issues other than the inbound email from the point in time of the snapshot being lost.
