Fedora Salt Master - New installation

DustinB3403

@scottalanmiller said in Fedora Salt Master - New installation:

@DustinB3403 said in Fedora Salt Master - New installation:

I just did pkill salt-master and am checking salt-master --log-level=debug and have more output this time.

You should not be killing or starting Salt manually. Use systemctl to start and stop. I think that's what is confusing you.

I just tried that as a step because the dev github recommended it for another person who is having the same exact issue.

Literally brand new install of Fed 30 Server.

@JaredBusch, the service was always running, the ports have never worked.

scottalanmiller

@DustinB3403 said in Fedora Salt Master - New installation:

@JaredBusch, the service was always running, the ports have never worked.

Something is wrong there. The thing that is determing that the "ports do not work" was based on trying to start the Salt Master, but it was already running. That tells us that the ports were working.

Other than that, why do you feel that the ports are not working?

DustinB3403

firewall-cmd --list-ports
4505/tcp 4506/tcp

DustinB3403

 firewall-cmd --list-ports
4505/tcp 4506/tcp
[root@localhost ~]# systemctl stop salt-master
[root@localhost ~]# systemctl stop salt-master.service
[root@localhost ~]# systemctl start salt-master.service
[root@localhost ~]# systemctl status salt-master.service
● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-08 16:31:42 EDT; 5s ago
	 Docs: man:salt-master(1)
		   file:///usr/share/doc/salt/html/contents.html
		   https://docs.saltstack.com/en/latest/contents.html
 Main PID: 3121 (salt-master)
	Tasks: 27 (limit: 2350)
   Memory: 387.5M
   CGroup: /system.slice/salt-master.service
		   ├─3121 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3128 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3130 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3131 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3134 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3135 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3136 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3137 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3139 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3145 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3146 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3147 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3150 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3333 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3334 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3335 /usr/bin/python2.7 /usr/bin/salt-master
		   ├─3336 /usr/bin/python2.7 /usr/bin/salt-master
		   └─3339 /usr/bin/python2.7 /usr/bin/salt-master

May 08 16:31:41 localhost.localdomain systemd[1]: Starting The Salt Master Server...
May 08 16:31:41 localhost.localdomain salt-master[3121]: /usr/lib/python2.7/site-packages/salt/scripts.py:102: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 >
May 08 16:31:42 localhost.localdomain systemd[1]: Started The Salt Master Server.
[root@localhost ~]# salt-master --log-level=debug
/usr/lib/python2.7/site-packages/salt/scripts.py:102: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Master
[WARNING ] Unable to bind socket master-ip:4505, error: [Errno 98] Address already in use; Is there another salt-master running?
[INFO    ] The Salt Master is shut down
[DEBUG   ] Stopping the multiprocessing logging queue listener
[DEBUG   ] closing multiprocessing queue
[DEBUG   ] joining multiprocessing queue thread
[DEBUG   ] Stopped the multiprocessing logging queue listener
The salt master is shutdown. The ports are not available to bind

DustinB3403

firewall-cmd --list-all
FedoraServer (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: cockpit dhcpv6-client ssh
  ports: 4505/tcp 4506/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

scottalanmiller

Do a netstat -tulpn again. Anytime you see that those ports are in use, that means that Salt-Master is running. Not sure what is starting it, but there is no question that it is being started somewhere. Only one copy can bind to the ports.

DustinB3403

@scottalanmiller said in Fedora Salt Master - New installation:

Do a netstat -tulpn again.

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      803/sshd
tcp        0      0 master-ip:4505     0.0.0.0:*               LISTEN      1019/python2.7
tcp        0      0 master-ip:4506     0.0.0.0:*               LISTEN      1025/python2.7
tcp6       0      0 :::22                   :::*                    LISTEN      803/sshd
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           749/chronyd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           916/dhclient
udp6       0      0 ::1:323                 :::*                                749/chronyd

DustinB3403

Finally!

salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
admins-MacBook-Air.local

coliver

@DustinB3403 said in Fedora Salt Master - New installation:

[root@localhost ~]# salt-master --log-level=debug

Doesn't this mean that you are starting the salt-master via the CLI and setting the log-level to debug? Stop the salt-master service with systemctl and rerun this command. I bet the output will be different.

JaredBusch

@coliver said in Fedora Salt Master - New installation:

@DustinB3403 said in Fedora Salt Master - New installation:

[root@localhost ~]# salt-master --log-level=debug

Doesn't this mean that you are starting the salt-master via the CLI and setting the log-level to debug? Stop the salt-master service with systemctl and rerun this command. I be the output will be different.

Correct. He is trying to start a second instance.

DustinB3403

@JaredBusch said in Fedora Salt Master - New installation:

Correct. He is trying to start a second instance.

Technically just following the official documentation. If it starts a second instance shame on them for not clarifying that point.

As the documentation reads, it's just to pull detailed logs.

scottalanmiller

@coliver said in Fedora Salt Master - New installation:

Doesn't this mean that you are starting the salt-master via the CLI

That's what I kept saying

scottalanmiller

@DustinB3403 said in Fedora Salt Master - New installation:

@JaredBusch said in Fedora Salt Master - New installation:

Correct. He is trying to start a second instance.

Technically just following the official documentation. If it starts a second instance shame on them for not clarifying that point.

As the documentation reads, it's just to pull detailed logs.

I read the doc, I don't see it ever saying to do that. Where did you get that command from?

DustinB3403

@scottalanmiller said in Fedora Salt Master - New installation:

Where did you get that command from?

https://docs.saltstack.com/en/2015.8/ref/configuration/index.html

coliver

Literally says:

It's pretty clear.

scottalanmiller

@coliver said in Fedora Salt Master - New installation:

Literally says:

It's pretty clear.

Ah, not part of the installation docs. That's why I didn't see it. Going through the installation, it is all set by the end of it, not sure where this configuration page pops in, but that's causing the problem. It was all installed and configured previously by following the installation guide.

Dashrender

@scottalanmiller said in Fedora Salt Master - New installation:

Do a netstat -tulpn again. Anytime you see that those ports are in use, that means that Salt-Master is running. Not sure what is starting it, but there is no question that it is being started somewhere. Only one copy can bind to the ports.

I was wondering if this was the issue - that Dustin is trying to start a second copy.

scottalanmiller

@Dashrender said in Fedora Salt Master - New installation:

@scottalanmiller said in Fedora Salt Master - New installation:

Do a netstat -tulpn again. Anytime you see that those ports are in use, that means that Salt-Master is running. Not sure what is starting it, but there is no question that it is being started somewhere. Only one copy can bind to the ports.

I was wondering if this was the issue - that Dustin is trying to start a second copy.

So this issue here is that there is an installation guide that gets you fully up and running. There there is a full reference for doing everything by hand. The first part was set up using the installation guide. Then that command to start the process manually was buried deep in the commands reference, disconnected from the installation guide. The first bit is a one page installer than is conclusive, a couple commands and you are done.

The configuration guide is for people doing everything by hand, such as writing your own start up scripts instead of using the ones from the installer. So the first part assumes a standard installation. The second part assumes you are constructing the system manually and haven't used the first part.

scottalanmiller

This is the installation page. You can see how short it is.

https://docs.saltstack.com/en/2015.8/topics/installation/fedora.html

scottalanmiller

Here is the problem, I think...

https://docs.saltstack.com/en/2015.8/topics/troubleshooting/master.html

So if you go through their massive manual, they have a troubleshooting section. And they tell you to run Master in the foreground for troubleshooting. They are assuming that you will stop it in the background first, but they never mention it. It's not part of the installation, but it probably should have a note about stopping any background processes first as easily a lot of people won't realize you can have both.