@scottalanmiller said in So You Lost Your ERP MSP?:

@marcinozga said in So You Lost Your ERP MSP?:

Most software have help -> about menu option and it should list software vendor.

If you can log in

But we know the ERP is Oracle NetSuite. But like everyone, they went through a reseller and lost the more important contacts.

Start looking at date stamps and polling peeps' memories for an approximate install date.

Then, get them to run a report on IT expenses in a 24-36 month period and look for the wart. It's going to be a big one.

@scottalanmiller said in So You Lost Your ERP MSP?:

@PhlipElder said in So You Lost Your ERP MSP?:

@scottalanmiller said in So You Lost Your ERP MSP?:

Not sure if this is funny, or a rant, or what. So we are the MSP for a firm and we do everything except their ERP support. I actually like this as ERP sucks and they use some ERP we don't know so cool, that's a perfect situation. It's web based so other than making sure Chrome is installed, up to date, and clean, we don't have to worry about the ERP.

Except one little problem. Upon implementation of the new ERP, the total disregard for the selection and implementation process is apparent and now it turns out, there is no one in the company who knows who the ERP vendor is or how to reach them. Or the MSP that supports them. So, we get ticket after ticket asking for help with the ERP and we are like "um, we don't know anything at all, call the support desk for the ERP" and they are like "sure, but... who is that?" And, of course, we were never told who it was. It's a browser based app, we don't need to interact with that support firm so we weren't introduced or given contacts.

So now the key application upon which the entire company operates is an unsupported black hole of disaster waiting to happen. This is the problem with going with little, unknown companies and keeping everything at arm's length.

There's no one specific failing. Just a general disregard for running the business, I guess. A bit of an "I can't even" here.

We have a few of those.

We go in and dig up any and all relevant info on the app/LoB so that we can at least be ready for disaster recovery. We set the expectations right from get-go.

If we end up supporting the apps, which we have in many cases, so be it. We bill for it as there's nothing wrong with picking up a few extra hours here and there. Part of setting the expectations is to define that we are not the front-line support for the App/LoB but are happy to help where we can.

It's complicated for us as we are flat rate and the ERP is the majority of the work. We'd love to pick it up, but it's not like we just have hours to bill, that would be handy. If that was the case we'd be voluntarily taking anything that we can. We don't even have logins, though.

Back Billing is a part of the contract. Any third party App/LoB that falls into our laps that is out of the scope of our support contract is billable.

The one really crappy app we support we don't have a logon for. Well, we do now since the owner has shared theirs along with their unlock admin code and so long as they are not logged in we can troubleshoot the app. Believe me, it's super duper crappy.

@scottalanmiller said in So You Lost Your ERP MSP?:

Not sure if this is funny, or a rant, or what. So we are the MSP for a firm and we do everything except their ERP support. I actually like this as ERP sucks and they use some ERP we don't know so cool, that's a perfect situation. It's web based so other than making sure Chrome is installed, up to date, and clean, we don't have to worry about the ERP.

Except one little problem. Upon implementation of the new ERP, the total disregard for the selection and implementation process is apparent and now it turns out, there is no one in the company who knows who the ERP vendor is or how to reach them. Or the MSP that supports them. So, we get ticket after ticket asking for help with the ERP and we are like "um, we don't know anything at all, call the support desk for the ERP" and they are like "sure, but... who is that?" And, of course, we were never told who it was. It's a browser based app, we don't need to interact with that support firm so we weren't introduced or given contacts.

So now the key application upon which the entire company operates is an unsupported black hole of disaster waiting to happen. This is the problem with going with little, unknown companies and keeping everything at arm's length.

There's no one specific failing. Just a general disregard for running the business, I guess. A bit of an "I can't even" here.

We have a few of those.

We go in and dig up any and all relevant info on the app/LoB so that we can at least be ready for disaster recovery. We set the expectations right from get-go.

If we end up supporting the apps, which we have in many cases, so be it. We bill for it as there's nothing wrong with picking up a few extra hours here and there. Part of setting the expectations is to define that we are not the front-line support for the App/LoB but are happy to help where we can.

@Fredtx said in Multiple Tombstoned DC's:

@notverypunny said in Multiple Tombstoned DC's:

@Fredtx does the isolated site still exist in Sites and Services? What's the plan for that location if the ideal end goal is to have the vpn tunnel down and no site to site connection? (apologies if this was already covered)

Yes, the site still exist. I'm just confused as to why the KCC is adding the connection to the link when there is no network connectivity to that site. From my understanding, the whole purpose of the KCC is to create connections with the best paths, which this one would NOT be the best path since there's no network connectivity.

Is the defunct site's subnet set up in Sites? That's what is going to need to be changed or removed.

@Fredtx said in Multiple Tombstoned DC's:

@PhlipElder

Sorry, I didn't mean links. I meant inbound partners. AKA "connections" when viewing in AD Sites and Services.

Yes. That's what I understood to be said there.

If there are replication links there that were automatically generated then at one time the good site's DCs were replicating with the offline site's DCs.

@Fredtx said in Multiple Tombstoned DC's:

@PhlipElder said in Multiple Tombstoned DC's:

Just how much change is there between then and now?

I don't know. It's been 8 months so I imagine there has been quite a bit of changes.

Also, just to confirm. The KCC ONLY creates site links for sites that have network connectivity, correct? My coworker seems to think that the Highlands server was never connected to those 6 sites, but from what I recall, they need to be connected or KCC would not have created those links. Again my theory is someone removed those vpn tunnels, or the Highlands DC was configured at our Fort Worth Hub site, and later shipped to Highlands.

If there are replication links (auto) then there was comms between the two site's DCs.

@Fredtx said in Multiple Tombstoned DC's:

@PhlipElder said in Multiple Tombstoned DC's:

What was happening for ADDS/DNS there anyway that there'd be that many tombstoned DCs? How did authentication happen?

My theory is the vpn tunnels were removed, and nobody checked if there was any kind of dependencies for those tunnels.

Below is the current setup.

The replication disconnection/issue happened at Highlands with 6 of it's inbound partners. The one's with the strikethrough

FortWorth -Replicates from Highlands
Highlands -Replicates from Toronto , Edmonton, Fort Worth, Nashua, York, Fresno, New Freedom, Oakland, Atlanta, Pewaukee
Toronto -Replicates from Fort Worth, Highlands, Nashua
Fresno -Replicates from Fort Worth, Highlands, Nashua, Toronto
Pewaukee -Replicates from Higlands
Nashua -Replicates from Edmonton, Oakland, Pewaukee, York, New Freedom, Atlanta, Toronto, Fort Worth, Highlands, Fresno
Oakland -Nashua, Highlands, Fort Worth
Atlanta -Replicates from Highlands, Fort Worth, Toronto
York -Replicates from Highlands, Fort Worth
NewFreedom -Replicates from Nashua, Highlands, Fort Worth
Edmonton -Replicates from Highlands, Toronto

Okay, with that amount of time ...
https://pmeijden.wordpress.com/2011/01/12/domain-replication-has-exceeded-the-tombstone-lifetime/

[QUOTE]
Another way to achieve this goal is to extend the Tombstone lifetime with ADSI Edit. You can find the option in CN=Configuration,DC=ForestRootDomainName,CN=Services and CN=Windows NT. Right click CN=Directory Service, and then click Properties. In the Attribute column, click tombstoneLifetime and change the value. Check the event log for the last successful replication date, this is very important in deciding the correct number of days. Beware that it is possible that objects that were removed are showing up in Active Directory again! You have to be sure that there aren’t that many changes in AD otherwise you can end up with a big mess.
[/QUOTE]
Emphasis mine.

Just how much change is there between then and now?

If there's a fair amount, then DCPromo -Force to remove ADDS/DNS from them and then DCPromo them back in after cleaning up the metadata, DNS, Sites, Trusts of any lingering bits and pieces.

Again, make sure there's a known good backup before starting.

@Fredtx said in Multiple Tombstoned DC's:

@PhlipElder said in Multiple Tombstoned DC's:

Is there a list of known devices that authenticate against those now defunct DCs? Are they still authenticating?

Most likely workstations are still authenticating. I don't have a list

Oiy. That's a mess.

@Fredtx said in Multiple Tombstoned DC's:

@PhlipElder said in Multiple Tombstoned DC's:

We've done this a few times where the work to remove the errant DCs was way more than flipping the bit, waiting and watching to make sure they don't screw anything up, and then flip the bit back.

Yea, I'm trying to be efficient as well, but also not screw anything up. lol. Especially since the Dc that has the fsmo roles also functions as Radius server for vpn and wireless authentication throughout different sites. I've only been here a month, so trying to get stuff working as it should.

Is there a list of known devices that authenticate against those now defunct DCs? Are they still authenticating?

One concern would be domain machine's passwords not being in sync with the PDCe.

What was happening for ADDS/DNS there anyway that there'd be that many tombstoned DCs? How did authentication happen?

@Fredtx said in Multiple Tombstoned DC's:

I've got 11 AD sites. 1 of the 11 has 6 inbound neighbors that have not replicated since 08/2021, possibly because someone deleted the VPN tunnels to those sites, and did not look at the dependencies of that tunnel.

I'm familiar with the demoting/promoting process, including the DNS cleanup that comes with it. My question is, do I need to demote all 6 of those inbound neighbors? Or is there a better way to handle this. I read that some people have had success with using the Lingering Object Liquidator (LoL) Microsoft tool, and forced AD replication by modifying the Allow replication with divergent and corrupt partner reg key.

You can flip the tombstone limit beyond the time they've been offline, give them a bit of time to get themselves caught up, then put the limit back to where it was before.

We've done this a few times where the work to remove the errant DCs was way more than flipping the bit, waiting and watching to make sure they don't screw anything up, and then flip the bit back.

Make sure to take a System State of your FSMO Role holder(s) before starting.

@nadnerB said in Random Thread - Anything Goes:

Must be Florida where 1:1,000,000 might use that stock off the left side of the column.

We're about 50/50 here in the Edmonton, Alberta region.

@voip_n00b said in Internal SMTP Relay:

Anyone have a good guide for setting up a internal smtp relay?

Exchange on-premises?

http://blog.mpecsinc.ca/2018/06/exchange-2013-set-up-receive-connector.html

That's the method we use to allow anonymous relay for devices on the network.

@scottalanmiller said in Random Thread - Anything Goes:

And you could make the fish talk to give status updates. Like it tells you when the Internet goes down or comes back. It suggests turning itself off and back on again. You could have Alexa built in!

OMG, I need investors, STAT.

And put a motion sensor under it so that when people walk by they hear the Mr. Creosote salutations.

EDIT: And if we're daring, "Fook OFF I'm Full" when reaching for a button!

@pete-s said in CentOS - What is the current opinion here?:

@scottalanmiller said in CentOS - What is the current opinion here?:

@pete-s said in CentOS - What is the current opinion here?:

I'm curious about what workloads you are thinking about.
I try but I can't think of any major application that doesn't run on both debian and redhat based distros.

Zimbra is one that always gets me. RHEL / CentOS/ Ubuntu LTS only. And they've tried to block CentOS in the past, but gave up on that.

OK, yeah, that seems to be one that is particularly sensitive.

IMHO if an application needs heavy integration into the OS and depends on specific package versions then it's better to turn the whole thing into an turn-key linux appliance. Like proxmox, xcp-ng, vyos, pfsense, freepbx, 3cx and other have done.

My guess is that Zimbra is getting by on mostly legacy installations though. Self-hosted email is hard to justify nowadays.

I like this.

I do think that there's some of this in a few different appliances that have hit the market over the last while in the Windows world. There is definitely a set it and forget it market out there though, in my mind, with a very important caveat that the appliance needs to be highly isolated for its specific purpose.

@scottalanmiller said in CentOS - What is the current opinion here?:

@phlipelder said in CentOS - What is the current opinion here?:

@travisdh1 said in CentOS - What is the current opinion here?:

@dashrender said in CentOS - What is the current opinion here?:

@scottalanmiller said in CentOS - What is the current opinion here?:

@pete-s said in CentOS - What is the current opinion here?:

@jaredbusch said in CentOS - What is the current opinion here?:

But this is the thing, not all applications are designed to run on various operating systems. So you do not always have the luxury.

That's true, you have to run what works. But most of the time you can stick to one OS.

Not that we've found. Finding an environment where you are running Linux, and can avoid all variation is pretty rare, I think. So many apps only work on Ubuntu XOR CentOS. It's a mes out there.

You can say that again.

Ok, I will, just look at my home lab! Fedora, Ubuntu and Debian because things either only run or run much better on different ones.

Say what one wills about Microsoft Windows, this conversation makes it clear that things are relatively homogeneous on that side of things. All things being equal and the company is not stuck on some encapsulated AS400 app or something.

It seems that way, but I feel like Linux apps tend to be almost always for current versions of the OS. At least production ones. But on Windows the amount of "we only support really old versions" is really high (of course, you can argue, anything that does that can't be production, right?)

But really, the issue is that Windows is an OS and Ubuntu is an OS. So there is no fracturing. The thing that gets weird is when you look at a large family of similar operating systems and compare them to the single OS of Windows. Sure, it seems fractured. That's because we are comparing an orchard to a tree. OS to OS, there is no fracturing.

There's no direct comparison. In the Windows world, though, you get "only licensed for Server" or "only runs on desktop" version issues that create similar fractures even on a single OS. We are constantly talking to clients about having gotten the wrong OS version for their needs. That never happens with Linux because there aren't those licensing limits or version in that way. It's a totally different style of problem, but it exists there, too.

A better analogy in my mind would be in grafting.

With Windows, we are essentially grafting onto the same trunk.

With *NIX that is not the case. Each distro has its own trunk and requires its own set of processes to graft on.

An OS is an OS is an OS. But, the apps/roles that run on top of that OS need Windows Server for server apps and Windows Desktop for desktop apps without getting into the Modern mess.

As I see it, that is not the case with *NIX / *BSD as each would require its own system of management, knowledge to install and and support both the OS and the apps, and then there's the post deployment support.

With support our preference is to be as homogeneous as possible. Having so many distros to manage and update instead of just one Windows OS with perhaps a few versions online looks to be a lot more complex.

Then, there's the whole University of Minnesota affair with the *NIX Kernel Team. Hypocrite Commits. SolarWinds anyone?

@travisdh1 said in CentOS - What is the current opinion here?:

@dashrender said in CentOS - What is the current opinion here?:

@scottalanmiller said in CentOS - What is the current opinion here?:

@pete-s said in CentOS - What is the current opinion here?:

@jaredbusch said in CentOS - What is the current opinion here?:

But this is the thing, not all applications are designed to run on various operating systems. So you do not always have the luxury.

That's true, you have to run what works. But most of the time you can stick to one OS.

Not that we've found. Finding an environment where you are running Linux, and can avoid all variation is pretty rare, I think. So many apps only work on Ubuntu XOR CentOS. It's a mes out there.

You can say that again.

Ok, I will, just look at my home lab! Fedora, Ubuntu and Debian because things either only run or run much better on different ones.

Say what one wills about Microsoft Windows, this conversation makes it clear that things are relatively homogeneous on that side of things. All things being equal and the company is not stuck on some encapsulated AS400 app or something.

@adamf said in CentOS - What is the current opinion here?:

So I have 1 server that needs migrated from CentOS8. What is the current state/opinion here about migration? Ubuntu is a clear choice, and most likely the path I will take, but wanted to get some other opinions as well. Is anyone using CentOS stream in a production scenario?

I've looked through here:
https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-rhel-centos-or-fedora

I don't see any specifications for cores/threads and storage requirements which I find odd? Is there a *NIX assumption or something about system resources?

Ah ... as I get closer to a more accurate set of search terms the search foo increases ...

https://docs.ansible.com/ansible-tower/2.2.2/html/installandreference/requirements_refguide.html

Anyway, my though/search strings aside, we're doing Mastodon on Ubuntu 20.04 LTS with not an issue other than tweaking scripts that were written for an earlier version of Ubuntu.

Ubuntu is my suggestion for an OS.

@jaredbusch said in Need audio cable help:

I have a digital piano that my daughter uses.

It has a 1/4 inch headphone jacks for output if desired.

I need to convert that to a mic input for a tablet on a standard mini jack.

She uses an app on her tablet that listens for her to play on the best correctly. But with the built in mic it misses notes often. The app recommends a direct connection to improve that.

I cannot find a cable for that. My search terms are simply returning various standard adapters. Standard adapters do not move the output audio to the mic input. Simply to the audio output of the new size.

As an alternative to mucking about with all of that, a simple YETI Blue microphone close to the piano's speaker would work.

They are phenomenal mics. It could then be used for her to pick up voice if she wants to get into recording and playing around with creating music.

EDIT: Add a FocusRite Scarlett and ProTools if she gets serious and a few YETI Pro series mics.

@scottalanmiller said in Is xByte still recommended for server purchases around here?:

@phlipelder said in Is xByte still recommended for server purchases around here?:

No black boxes for critical data. Ever.

This is a good quote.

Should be, for critical workloads, though. Not only storage, but any component of it.

It's why we've been rolling our own for close to two decades now.

When we do Tier 1 we get Tier 1 4 hour response with that along with an extention to 60 months.

I still prefer our own over the others.

@scottalanmiller said in Is xByte still recommended for server purchases around here?:

@phlipelder said in Is xByte still recommended for server purchases around here?:

When the shit hits the fan, I want real support with real people. That's gonna cost more than some box with a baby motherboard, some memory, some sort of flash storage for the *NIX OS, and whoever's drives in the drive bay.

That's the thing, it's Linux, so the top enterprise support is available. That you got it from a NAS vendor doesn't really matter. Sure, it would be BETTER with better hardware and a more enterprise version of Linux. I'm not saying it doesn't get better. But you can get all that enterprise support in any low end NAS box if you want, because the components being supported are universal.

Just like if you were to put Windows on a Dell server... the quality of enterprise level support comes down to who can support Windows. That it's on Dell or a piece of crap hardware might make a tiny difference for hardware uptime, but has no bearing on the quality of support that matters. The NAS box can be replaced for a Dell, HPE, Cisco, whatever big "support" brand you want after data loss if you want. The hardware is actually interchangeable here.

That hasn't been my experience with any of the NAS vendors.

Even the Synology 2U NAS/SAN to NAS/SAN replication units that were supposed to be transparent to the Hyper-V cluster running in front of them. Synology refused to address our concerns with forum's posts that showed the promise was never fulfilled.

We won't ever deploy a NAS for anything critical. Backups are critical.

This is one of our chassis starting places the CS381:
https://www.silverstonetek.com/product.php?pid=861&area=en

It's an excellent platform. Their SFX power supplies are good. There are twin PSU setups out there but they are ATX so won't fit.

Micro-ATX offers a plethora of AMD EPYC, Intel Xeon Scalable, Intel Xeon, and other platforms to install in the box.

We have MLNX 25GbE RDMA running on both AMD EPYC and Intel Scalable platforms into SATA SSD for cache/journal and SATA or SAS NearLine spindles for bulk storage.

Our payback is really good with these boxes on 14TB to 16TB spindles.

We know what's in the box. We know how to fix things or recover things if thing go awry which they can. That's our comfort zone. No black boxes for critical data. Ever.