Best posts made by dgingerich

@scottalanmiller I have tried repeatedly to delete all keys and rejoin the minions. It doesn't work. The keys are seen and accepted, but then the minions refuse to authenticate. I have even deleted all keys, uninstalled salt from the minions, deleted all cached data, reinstalled salt minion, and resubmitted keys, and still, trying a ping to all minions right after accpeting the keys results in no connections. running salt-minion -l debug shows that the minions think the masters' keys don't authenticate. It is very frustrating.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value

maybe you have duplicate DNS entries and round robin is getting you?

The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.

Oh okay, this is all hosted. Still, best to be sure and rule out possibilities while testing. This is weird, we use Vultr for Salt Masters and have never seen anything like this. But we avoid Ubuntu, so if there is any bug there, we'd not have seen it.

I was able to build another system, named QAICS-mastertest, that worked perfectly using the exact same methods. It's really weird.

OK. I've deleted the DNS entries and the systems from Packet.net. I'm going to try again tomorrow morning after all the DNS caching should have expired.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

There's a lot of customization in the iptables config and other areas that are scripted so that future systems could be deployed quickly.

If that was handled in Salt, it would be all automated so that it would solve this problem, rather than create it. That's actually a reason to use Salt in that case.

The production stack has Salt working. It's the QA stack that doesn't. The production stack is 62 servers, and needs Salt. The QA stack is just 8 servers plus 2 management servers, and the salt config is supposed to be the same, but adjusted for other system names. I don't have the Salt skills to adjust the Salt scripts to work for the QA stack.

Hmmm... might want to pressure your Salt guy to automate that. In theory, you could have something like QA in the name that designates that for the future.

Yeah, the reason I'm in the middle of all this is because our Salt guy is on his honeymoon in Italy this week, and the UK next week. So, he's not going to be of much help for a while.

That was not good timing!

Fun story: He's planned this for a year. He married his wife a year ago in the US to allow her to stay in the country, to then have a real ceremony on their 1 year anniversary with her family and go on their honeymoon. In January, our company got bought out, specifically because our technology was far better than the other company's, yet they had better sales contacts and more business, and the new CTO has a very special and fast timeline for converting some of our apps to cloud for use by our new coworkers. That plan just happened to step on our Systems Engineer's wedding plans that were not able to be changed. So, we got stuck in this mess. Sometimes, various things just make the perfect storm.