Multiple Tombstoned DC's
-
I've got 11 AD sites. 1 of the 11 has 6 inbound neighbors that have not replicated since 08/2021, possibly because someone deleted the VPN tunnels to those sites, and did not look at the dependencies of that tunnel.
I'm familiar with the demoting/promoting process, including the DNS cleanup that comes with it. My question is, do I need to demote all 6 of those inbound neighbors? Or is there a better way to handle this. I read that some people have had success with using the Lingering Object Liquidator (LoL) Microsoft tool, and forced AD replication by modifying the Allow replication with divergent and corrupt partner reg key.
-
@Fredtx said in Multiple Tombstoned DC's:
I've got 11 AD sites. 1 of the 11 has 6 inbound neighbors that have not replicated since 08/2021, possibly because someone deleted the VPN tunnels to those sites, and did not look at the dependencies of that tunnel.
I'm familiar with the demoting/promoting process, including the DNS cleanup that comes with it. My question is, do I need to demote all 6 of those inbound neighbors? Or is there a better way to handle this. I read that some people have had success with using the Lingering Object Liquidator (LoL) Microsoft tool, and forced AD replication by modifying the Allow replication with divergent and corrupt partner reg key.
You can flip the tombstone limit beyond the time they've been offline, give them a bit of time to get themselves caught up, then put the limit back to where it was before.
We've done this a few times where the work to remove the errant DCs was way more than flipping the bit, waiting and watching to make sure they don't screw anything up, and then flip the bit back.
Make sure to take a System State of your FSMO Role holder(s) before starting.
-
@Fredtx said in Multiple Tombstoned DC's:
ot 11 AD sites. 1 of the 11 has 6 inbound neighbors that have not replicated since 08/2021, possibly because someone deleted the VPN tunnels to those sites, and did not look at the dependencies of that tunnel.
I'm familiar with the demoting/promoting process, including the DNS cleanup that comes with it. My question is, do I need to demote all 6 of those inbound neighbors? Or is there a better way to handle this. I read that some people have had success with using the Lingering Object Liquidator (LoL) Microsoft tool, and forced AD replication by modifying the Allow replication with divergent and corrupt partner reg key.I would demote them fully and then add them slowly each one to make sure they are being added properly to the domain.
-
@dbeato said in Multiple Tombstoned DC's:
@Fredtx said in Multiple Tombstoned DC's:
ot 11 AD sites. 1 of the 11 has 6 inbound neighbors that have not replicated since 08/2021, possibly because someone deleted the VPN tunnels to those sites, and did not look at the dependencies of that tunnel.
I'm familiar with the demoting/promoting process, including the DNS cleanup that comes with it. My question is, do I need to demote all 6 of those inbound neighbors? Or is there a better way to handle this. I read that some people have had success with using the Lingering Object Liquidator (LoL) Microsoft tool, and forced AD replication by modifying the Allow replication with divergent and corrupt partner reg key.I would demote them fully and then add them slowly each one to make sure they are being added properly to the domain.
I was going to do 1 at a time, but ran into issues with the 1st as one of the other tombstone DC accepted the logon of the server when I joined it back to domain, so I had problems trying to promote it as I was pointing it to our corporate site, which corp site did not have the new computer object. I plan on demoting all 6 at one time Friday night, but was thinking there could be a better way to handle this possibly, or what other options I have.
-
@PhlipElder said in Multiple Tombstoned DC's:
We've done this a few times where the work to remove the errant DCs was way more than flipping the bit, waiting and watching to make sure they don't screw anything up, and then flip the bit back.
Yea, I'm trying to be efficient as well, but also not screw anything up. lol. Especially since the Dc that has the fsmo roles also functions as Radius server for vpn and wireless authentication throughout different sites. I've only been here a month, so trying to get stuff working as it should.
-
@Fredtx said in Multiple Tombstoned DC's:
@PhlipElder said in Multiple Tombstoned DC's:
We've done this a few times where the work to remove the errant DCs was way more than flipping the bit, waiting and watching to make sure they don't screw anything up, and then flip the bit back.
Yea, I'm trying to be efficient as well, but also not screw anything up. lol. Especially since the Dc that has the fsmo roles also functions as Radius server for vpn and wireless authentication throughout different sites. I've only been here a month, so trying to get stuff working as it should.
Is there a list of known devices that authenticate against those now defunct DCs? Are they still authenticating?
One concern would be domain machine's passwords not being in sync with the PDCe.
What was happening for ADDS/DNS there anyway that there'd be that many tombstoned DCs? How did authentication happen?
-
@PhlipElder said in Multiple Tombstoned DC's:
Is there a list of known devices that authenticate against those now defunct DCs? Are they still authenticating?
Most likely workstations are still authenticating. I don't have a list
-
@Fredtx said in Multiple Tombstoned DC's:
@PhlipElder said in Multiple Tombstoned DC's:
Is there a list of known devices that authenticate against those now defunct DCs? Are they still authenticating?
Most likely workstations are still authenticating. I don't have a list
Oiy. That's a mess.
-
@PhlipElder said in Multiple Tombstoned DC's:
What was happening for ADDS/DNS there anyway that there'd be that many tombstoned DCs? How did authentication happen?
My theory is the vpn tunnels were removed, and nobody checked if there was any kind of dependencies for those tunnels.
Below is the current setup.
The replication disconnection/issue happened at Highlands with 6 of it's inbound partners. The one's with the strikethrough
FortWorth -Replicates from Highlands
Highlands -Replicates fromToronto, Edmonton, Fort Worth,Nashua,York,Fresno,New Freedom,Oakland, Atlanta, Pewaukee
Toronto -Replicates from Fort Worth, Highlands, Nashua
Fresno -Replicates from Fort Worth, Highlands, Nashua, Toronto
Pewaukee -Replicates from Higlands
Nashua -Replicates from Edmonton, Oakland, Pewaukee, York, New Freedom, Atlanta, Toronto, Fort Worth, Highlands, Fresno
Oakland -Nashua, Highlands, Fort Worth
Atlanta -Replicates from Highlands, Fort Worth, Toronto
York -Replicates from Highlands, Fort Worth
NewFreedom -Replicates from Nashua, Highlands, Fort Worth
Edmonton -Replicates from Highlands, Toronto -
@Fredtx said in Multiple Tombstoned DC's:
@PhlipElder said in Multiple Tombstoned DC's:
What was happening for ADDS/DNS there anyway that there'd be that many tombstoned DCs? How did authentication happen?
My theory is the vpn tunnels were removed, and nobody checked if there was any kind of dependencies for those tunnels.
Below is the current setup.
The replication disconnection/issue happened at Highlands with 6 of it's inbound partners. The one's with the strikethrough
FortWorth -Replicates from Highlands
Highlands -Replicates fromToronto, Edmonton, Fort Worth,Nashua,York,Fresno,New Freedom,Oakland, Atlanta, Pewaukee
Toronto -Replicates from Fort Worth, Highlands, Nashua
Fresno -Replicates from Fort Worth, Highlands, Nashua, Toronto
Pewaukee -Replicates from Higlands
Nashua -Replicates from Edmonton, Oakland, Pewaukee, York, New Freedom, Atlanta, Toronto, Fort Worth, Highlands, Fresno
Oakland -Nashua, Highlands, Fort Worth
Atlanta -Replicates from Highlands, Fort Worth, Toronto
York -Replicates from Highlands, Fort Worth
NewFreedom -Replicates from Nashua, Highlands, Fort Worth
Edmonton -Replicates from Highlands, TorontoOkay, with that amount of time ...
https://pmeijden.wordpress.com/2011/01/12/domain-replication-has-exceeded-the-tombstone-lifetime/[QUOTE]
Another way to achieve this goal is to extend the Tombstone lifetime with ADSI Edit. You can find the option in CN=Configuration,DC=ForestRootDomainName,CN=Services and CN=Windows NT. Right click CN=Directory Service, and then click Properties. In the Attribute column, click tombstoneLifetime and change the value. Check the event log for the last successful replication date, this is very important in deciding the correct number of days. Beware that it is possible that objects that were removed are showing up in Active Directory again! You have to be sure that there aren’t that many changes in AD otherwise you can end up with a big mess.
[/QUOTE]
Emphasis mine.Just how much change is there between then and now?
If there's a fair amount, then DCPromo -Force to remove ADDS/DNS from them and then DCPromo them back in after cleaning up the metadata, DNS, Sites, Trusts of any lingering bits and pieces.
Again, make sure there's a known good backup before starting.
-
@PhlipElder said in Multiple Tombstoned DC's:
Just how much change is there between then and now?
I don't know. It's been 8 months so I imagine there has been quite a bit of changes.
Also, just to confirm. The KCC ONLY creates site links for sites that have network connectivity, correct? My coworker seems to think that the Highlands server was never connected to those 6 sites, but from what I recall, they need to be connected or KCC would not have created those links. Again my theory is someone removed those vpn tunnels, or the Highlands DC was configured at our Fort Worth Hub site, and later shipped to Highlands.
-
@Fredtx said in Multiple Tombstoned DC's:
@PhlipElder said in Multiple Tombstoned DC's:
Just how much change is there between then and now?
I don't know. It's been 8 months so I imagine there has been quite a bit of changes.
Also, just to confirm. The KCC ONLY creates site links for sites that have network connectivity, correct? My coworker seems to think that the Highlands server was never connected to those 6 sites, but from what I recall, they need to be connected or KCC would not have created those links. Again my theory is someone removed those vpn tunnels, or the Highlands DC was configured at our Fort Worth Hub site, and later shipped to Highlands.
If there are replication links (auto) then there was comms between the two site's DCs.
-
Sorry, I didn't mean links. I meant inbound partners. AKA "connections" when viewing in AD Sites and Services.
-
@Fredtx said in Multiple Tombstoned DC's:
Sorry, I didn't mean links. I meant inbound partners. AKA "connections" when viewing in AD Sites and Services.
Yes. That's what I understood to be said there.
If there are replication links there that were automatically generated then at one time the good site's DCs were replicating with the offline site's DCs.
-
@PhlipElder said in Multiple Tombstoned DC's:
@Fredtx said in Multiple Tombstoned DC's:
Sorry, I didn't mean links. I meant inbound partners. AKA "connections" when viewing in AD Sites and Services.
Yes. That's what I understood to be said there.
If there are replication links there that were automatically generated then at one time the good site's DCs were replicating with the offline site's DCs.
I have demoted all defective sites, did metadata and dns clean up, confirmed replication of changes across all domain controllers.
I promoted the first dc successfully. However, the KCC is automatically adding the site that has no network connectivity, which I can't seem to understand why. This is why the dc's tombstoned in the first place. I guess I could create the vpn tunnel? I would think the KCC would detect there is no connectivity, and add a connection that DOES have connectivity.
Any ideas @dbeato @PhlipElder ???
-
@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)
-
@notverypunny said in Multiple Tombstoned DC's:
@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)
Yes, the site still exist. I'm just confused as to why the KCC is adding the connection to the link when there is no network connectivity to that site. From my understanding, the whole purpose of the KCC is to create connections with the best paths, which this one would NOT be the best path since there's no network connectivity.
-
@Fredtx said in Multiple Tombstoned DC's:
@notverypunny said in Multiple Tombstoned DC's:
@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)
Yes, the site still exist. I'm just confused as to why the KCC is adding the connection to the link when there is no network connectivity to that site. From my understanding, the whole purpose of the KCC is to create connections with the best paths, which this one would NOT be the best path since there's no network connectivity.
Is the defunct site's subnet set up in Sites? That's what is going to need to be changed or removed.
-
@PhlipElder said in Multiple Tombstoned DC's:
@Fredtx said in Multiple Tombstoned DC's:
@notverypunny said in Multiple Tombstoned DC's:
@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)
Yes, the site still exist. I'm just confused as to why the KCC is adding the connection to the link when there is no network connectivity to that site. From my understanding, the whole purpose of the KCC is to create connections with the best paths, which this one would NOT be the best path since there's no network connectivity.
Is the defunct site's subnet set up in Sites? That's what is going to need to be changed or removed.
This is where my idea was headed, but wanted make sure that the OP realized that without AD connectivity it's going to be entirely off the domain and that his other domain machines are going to tombstone as well (if they haven't already)
-
@PhlipElder said in Multiple Tombstoned DC's:
@Fredtx said in Multiple Tombstoned DC's:
@notverypunny said in Multiple Tombstoned DC's:
@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)
Yes, the site still exist. I'm just confused as to why the KCC is adding the connection to the link when there is no network connectivity to that site. From my understanding, the whole purpose of the KCC is to create connections with the best paths, which this one would NOT be the best path since there's no network connectivity.
Is the defunct site's subnet set up in Sites? That's what is going to need to be changed or removed.
Yes, it's setup in Sites with the correct subnet as well.
Also, I see all the sites are listed in the Sites/Inter-Site Transports/IP/Defaultipsite link properties. Is that normal? Highlands (defunct site) does have the appropriate subnet configured as I mentioned.
See below
