Unsolved Salt Minion Problems

[Romo]

@scottalanmiller shouldn't the salt-master have already sent its own key? In my vm the first key I accepted was the key my master.

This could mean there is something else that's wrong don't you think?

[scottalanmiller]

@Romo said in Salt Minion Problems:

@scottalanmiller shouldn't the salt-master have already sent its own key? In my vm the first key I accepted was the key my master.

This could mean there is something else that's wrong don't you think?

Which part? The master? That seems fine.

[Romo]

@scottalanmiller Yes, on my install the master shows fine on @WrCombs the master doesnt appear.

[scottalanmiller]

Don't rename the hosts, that's going to lead to more problems.

[scottalanmiller]

I'm rebooting to test something. For some reason, the Salt Master never showed up in the key list either. This implies something is certainly wrong.

[Romo]

@scottalanmiller said in Salt Minion Problems:

I'm rebooting to test something. For some reason, the Salt Master never showed up in the key list either. This implies something is certainly wrong.

Exactly thats what I ment

@scottalanmiller said in Salt Minion Problems:

@Romo said in Salt Minion Problems:

@scottalanmiller shouldn't the salt-master have already sent its own key? In my vm the first key I accepted was the key my master.

This could mean there is something else that's wrong don't you think?

Which part? The master? That seems fine.

[Romo]

@WrCombs By the way, do you remember if you did open the ports in the master?

[Romo]

All set on my install, I hadn't reloaded the firewall rules so that's why my minion could reach the master.

[scottalanmiller]

@Romo said in Salt Minion Problems:

@WrCombs By the way, do you remember if you did open the ports in the master?

He did, I looked.

[scottalanmiller]

Here is his firewall...

# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client ssh
  ports: 4505-4506/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules: 

[scottalanmiller]

Here is the error message on the Salt Master's minion process:

Jan 31 20:32:37 salt salt-minion[901]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

[Romo]

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

[scottalanmiller]

@Romo said in Salt Minion Problems:

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

Yeah, the master didn't have an /etc/hosts entry so it was pointing itself to a different Salt Master. So that was awfully confusing.

[scottalanmiller]

Ok, so after fixing the hostname on the master, fixing the /etc/hosts file and then checking the errors, we find that that weird ip local address there is NOT from the Minion, but from the Master itself. So the Minion has never checked in at all.

# salt-key --list-all
Accepted Keys:
Denied Keys:
ip-65-75-137-152.local
Unaccepted Keys:
ip-65-75-137-152.local
Rejected Keys:
[root@lab-lnx-william-salt ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2017-01-31 20:39:37 UTC; 2min 43s ago
  Process: 902 ExecStart=/usr/bin/salt-minion (code=exited, status=0/SUCCESS)
 Main PID: 902 (code=exited, status=0/SUCCESS)

Jan 31 20:39:22 lab-lnx-william-salt systemd[1]: Starting The Salt Minion...
Jan 31 20:39:23 lab-lnx-william-salt systemd[1]: Started The Salt Minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: [CRITICAL] The Salt Master has rejected this minion's public key!
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

[scottalanmiller]

Here is the current error on the minion. I'm guessing we have the same issue:

[root@minion1 ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-01-31 20:21:59 UTC; 23min ago
 Main PID: 1228 (salt-minion)
   CGroup: /system.slice/salt-minion.service
           ├─1228 /usr/bin/python /usr/bin/salt-minion
           ├─1237 /usr/bin/python /usr/bin/salt-minion
           └─1241 /usr/bin/python /usr/bin/salt-minion

Jan 31 20:44:27 minion1 salt-minion[1228]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

[scottalanmiller]

Here are some issues in the /etc/hosts file that will definitely cause things to fail:

192.168.1.97



192.168.1.94 salt
192.128.1.94 salt

[Dashrender]

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

[scottalanmiller]

@Dashrender said in Salt Minion Problems:

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

No, you would need this to be correct either way.

[Romo]

@scottalanmiller check what does salt have as the minion_id as well, it is in /etc/salt/minion_id

[scottalanmiller]

Also found the /etc/hosts file had a full double copy of itself in the same file. No idea how that got there.