Unsolved Salt Minion Problems
-
@WrCombs By the way, do you remember if you did open the ports in the master?
-
All set on my install, I hadn't reloaded the firewall rules so that's why my minion could reach the master.
-
@Romo said in Salt Minion Problems:
@WrCombs By the way, do you remember if you did open the ports in the master?
He did, I looked.
-
Here is his firewall...
# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: dhcpv6-client ssh ports: 4505-4506/tcp protocols: masquerade: no forward-ports: sourceports: icmp-blocks: rich rules: -
Here is the error message on the Salt Master's minion process:
Jan 31 20:32:37 salt salt-minion[901]: [ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate -
@scottalanmiller said in Salt Minion Problems:
The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
That happens to all minions when they are waiting for the master to approve their key.
-
@Romo said in Salt Minion Problems:
@scottalanmiller said in Salt Minion Problems:
The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
That happens to all minions when they are waiting for the master to approve their key.
Yeah, the master didn't have an /etc/hosts entry so it was pointing itself to a different Salt Master. So that was awfully confusing.
-
Ok, so after fixing the hostname on the master, fixing the /etc/hosts file and then checking the errors, we find that that weird ip local address there is NOT from the Minion, but from the Master itself. So the Minion has never checked in at all.
# salt-key --list-all Accepted Keys: Denied Keys: ip-65-75-137-152.local Unaccepted Keys: ip-65-75-137-152.local Rejected Keys: [root@lab-lnx-william-salt ~]# systemctl status salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Active: inactive (dead) since Tue 2017-01-31 20:39:37 UTC; 2min 43s ago Process: 902 ExecStart=/usr/bin/salt-minion (code=exited, status=0/SUCCESS) Main PID: 902 (code=exited, status=0/SUCCESS) Jan 31 20:39:22 lab-lnx-william-salt systemd[1]: Starting The Salt Minion... Jan 31 20:39:23 lab-lnx-william-salt systemd[1]: Started The Salt Minion. Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: [CRITICAL] The Salt Master has rejected this minion's public key! Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: To repair this issue, delete the public key for this minion on the Salt Master and restart this minion. Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit. -
Here is the current error on the minion. I'm guessing we have the same issue:
[root@minion1 ~]# systemctl status salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-01-31 20:21:59 UTC; 23min ago Main PID: 1228 (salt-minion) CGroup: /system.slice/salt-minion.service ├─1228 /usr/bin/python /usr/bin/salt-minion ├─1237 /usr/bin/python /usr/bin/salt-minion └─1241 /usr/bin/python /usr/bin/salt-minion Jan 31 20:44:27 minion1 salt-minion[1228]: [ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate -
Here are some issues in the /etc/hosts file that will definitely cause things to fail:
192.168.1.97 192.168.1.94 salt 192.128.1.94 salt -
All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol
-
@Dashrender said in Salt Minion Problems:
All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol
No, you would need this to be correct either way.
-
@scottalanmiller check what does salt have as the minion_id as well, it is in /etc/salt/minion_id
-
Also found the /etc/hosts file had a full double copy of itself in the same file. No idea how that got there.
-
Salt Master will not allow me to delete the keys that are in there. I did a delete all and it said that it was, but it failed.
-
okay. what do i need to do to fix it? sorry for the headache guys
-
well it looks like Scott fixed most of the problems. ( ish ) I just logged on to the server and all the things he pointed out as wrong, arent there anymore
-
@WrCombs said in Salt Minion Problems:
okay. what do i need to do to fix it? sorry for the headache guys
No idea.
-
@scottalanmiller said in Salt Minion Problems:
@Dashrender said in Salt Minion Problems:
All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol
No, you would need this to be correct either way.
I'm sorry - correct what either way?
If he had no DNS entry for salt, sure it would fail and he would be told to put entries in /etc/hosts but since there was a working salt server in DNS, things appear to work, but really aren't going where expected...
So having an over lapping lab just made the situation harder - ultimately learning more I guess.
-
@WrCombs said in Salt Minion Problems:
well it looks like Scott fixed most of the problems. ( ish ) I just logged on to the server and all the things he pointed out as wrong, arent there anymore
Don't be - this is learning, sometimes you break things. Sometimes you don't. Scott digging into what you were doing exactly can be helpful in making guides that help future people learn with a little less pain.
@scottalanmiller can you pull a log of all of the commands he ran and perhaps find things like how the hosts file was completely doubled?
