HAProxy fails to start on reboot



  • Can someone explain why my system refuses to start the HAProxy service on reboot?

    But if I subsequently log in and issue systemctl start haproxy it works?

    Looking at journalctl seems to point to a problem attaching to some ports initially.

    -- Reboot --
    Feb 25 21:36:42 hap.domain.local systemd[1]: Starting HAProxy Load Balancer...
    Feb 25 21:36:43 hap.domain.local systemd[1]: Started HAProxy Load Balancer.
    Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_stats: cannot bind socket [10.202.0.20:8080]
    Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_exchange_https: cannot bind socket [10.202.0.19:80]
    Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_exchange_https: cannot bind socket [10.202.0.19:443]
    Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: haproxy-systemd-wrapper: exit, haproxy RC=1
    Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE
    Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Unit entered failed state.
    Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Failed with result 'exit-code'.
    Feb 25 21:41:24 hap.domain.local systemd[1]: Starting HAProxy Load Balancer...
    Feb 25 21:41:24 hap.domain.local systemd[1]: Started HAProxy Load Balancer.
     
    


  • My guesses would be that for some reason it is trying to fire up as a non-root user, or SELinux is messing with it, but then changing later so as not to be a problem.



  • @scottalanmiller said in HAProxy fails to start on reboot:

    My guesses would be that for some reason it is trying to fire up as a non-root user,

    As is typical for my client work, everything is done as the root user to enable simpler hand off in case we part ways.

    I manage things with my own user account and sudo, but the setup is all originally done under root.

    So because of that, I cannot see what would be causing this from that point of view.

    @scottalanmiller said in HAProxy fails to start on reboot:

    or SELinux is messing with it, but then changing later so as not to be a problem.

    A possibility I guess. I can disable it permanently, reboot, and see what happens as a test.



  • set it to permissive in /etc/selinux/config and rebooted. same result.

    this be annoying.



  • @jaredbusch said in HAProxy fails to start on reboot:

    set it to permissive in /etc/selinux/config and rebooted. same result.

    this be annoying.

    Well at least that is ruled out.



  • This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?



  • @scottalanmiller said in HAProxy fails to start on reboot:

    This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

    Single purpose VM, so should not be. Especially on 8080.



  • @jaredbusch said in HAProxy fails to start on reboot:

    @scottalanmiller said in HAProxy fails to start on reboot:

    This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

    Single purpose VM, so should not be. Especially on 8080.

    netstat -apn after reboot?



  • I saw that someone needed this. But should not affect you with SELinux off.

    setsebool -P haproxy_connect_any=1


  • @scottalanmiller said in HAProxy fails to start on reboot:

    I saw that someone needed this. But should not affect you with SELinux off.

    setsebool -P haproxy_connect_any=1
    

    And it is running with it enforcing



  • What do you get with ip addr | grep 10



  • Llooks like Nginx is there, but there is no conf for it and no standard /etc/nginx folder.

    WTF, wonder if HAProxy uses it.



  • @jaredbusch said in HAProxy fails to start on reboot:

    Llooks like Nginx is there, but there is no conf for it and no standard /etc/nginx folder.

    WTF, wonder if HAProxy uses it.

    That's really odd. I didn't think that it used that. I think HA-Proxy is older than Nginx, actually.



  • @scottalanmiller said in HAProxy fails to start on reboot:

    What do you get with ip addr | grep 10

    shows the vif that the instructions had me setup.

    # ip addr | grep 10
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        inet 10.202.0.20/23 brd 10.202.1.255 scope global dynamic eth0
        inet 10.202.0.19/23 brd 10.202.1.255 scope global secondary eth0:0
    


  • That looks like it should.



  • @tim_g said in HAProxy fails to start on reboot:

    @jaredbusch said in HAProxy fails to start on reboot:

    @scottalanmiller said in HAProxy fails to start on reboot:

    This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

    Single purpose VM, so should not be. Especially on 8080.

    netstat -apn after reboot?

    $ sudo netstat -apn
    [sudo] password for jbusch: 
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      660/sshd            
    tcp        0    356 10.202.0.20:22          10.254.103.22:36038     ESTABLISHED 855/sshd: jbusch [p 
    tcp6       0      0 :::22                   :::*                    LISTEN      660/sshd            
    udp        0      0 127.0.0.1:323           0.0.0.0:*                           617/chronyd         
    udp        0      0 0.0.0.0:68              0.0.0.0:*                           800/dhclient        
    udp6       0      0 ::1:323                 :::*                                617/chronyd         
    raw6       0      0 :::58                   :::*                    7           638/NetworkManager  
    Active UNIX domain sockets (servers and established)
    Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
    unix  2      [ ACC ]     STREAM     LISTENING     17962    1/systemd            /run/lvm/lvmpolld.socket
    unix  2      [ ACC ]     SEQPACKET  LISTENING     17965    1/systemd            /run/udev/control
    unix  2      [ ]         DGRAM                    20289    617/chronyd          /var/run/chrony/chronyd.sock
    unix  2      [ ACC ]     SEQPACKET  LISTENING     18047    1/systemd            /run/systemd/coredump
    unix  3      [ ]         DGRAM                    11927    1/systemd            /run/systemd/notify
    unix  2      [ ]         DGRAM                    11929    1/systemd            /run/systemd/cgroups-agent
    unix  2      [ ACC ]     STREAM     LISTENING     18087    1/systemd            /run/lvm/lvmetad.socket
    unix  14     [ ]         DGRAM                    11944    1/systemd            /run/systemd/journal/dev-log
    unix  2      [ ACC ]     STREAM     LISTENING     11948    1/systemd            /run/systemd/journal/stdout
    unix  6      [ ]         DGRAM                    11950    1/systemd            /run/systemd/journal/socket
    unix  2      [ ACC ]     STREAM     LISTENING     22988    638/NetworkManager   /var/run/NetworkManager/private-dhcp
    unix  2      [ ACC ]     STREAM     LISTENING     20173    1/systemd            /run/dbus/system_bus_socket
    unix  2      [ ]         DGRAM                    23506    858/systemd          /run/user/1000/systemd/notify
    unix  2      [ ACC ]     STREAM     LISTENING     23508    858/systemd          /run/user/1000/systemd/private
    unix  2      [ ACC ]     STREAM     LISTENING     23514    858/systemd          /run/user/1000/bus
    unix  2      [ ACC ]     STREAM     LISTENING     17885    1/systemd            /run/systemd/private
    unix  3      [ ]         STREAM     CONNECTED     23561    855/sshd: jbusch [p  
    unix  3      [ ]         STREAM     CONNECTED     20320    615/dbus-daemon      
    unix  2      [ ]         DGRAM                    20583    625/hypervkvpd       
    unix  2      [ ]         DGRAM                    18222    472/systemd-journal  
    unix  2      [ ]         DGRAM                    23020    800/dhclient         
    unix  3      [ ]         STREAM     CONNECTED     20386    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     20006    591/auditd           
    unix  2      [ ]         DGRAM                    23552    855/sshd: jbusch [p  
    unix  2      [ ]         DGRAM                    23652    889/sudo             
    unix  3      [ ]         STREAM     CONNECTED     23560    864/sshd: [email protected]  
    unix  3      [ ]         STREAM     CONNECTED     20432    624/systemd-logind   
    unix  2      [ ]         DGRAM                    20598    624/systemd-logind   
    unix  3      [ ]         STREAM     CONNECTED     20319    615/dbus-daemon      
    unix  3      [ ]         STREAM     CONNECTED     20524    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     20433    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     20385    623/hypervvssd       
    unix  2      [ ]         DGRAM                    20575    623/hypervvssd       
    unix  3      [ ]         STREAM     CONNECTED     20005    591/auditd           
    unix  3      [ ]         STREAM     CONNECTED     20609    624/systemd-logind   
    unix  3      [ ]         STREAM     CONNECTED     20523    625/hypervkvpd       
    unix  2      [ ]         DGRAM                    20004    591/auditd           
    unix  2      [ ]         DGRAM                    22822    622/python3          
    unix  3      [ ]         STREAM     CONNECTED     18662    500/systemd-udevd    
    unix  3      [ ]         STREAM     CONNECTED     23470    472/systemd-journal  /run/systemd/journal/stdout
    unix  2      [ ]         DGRAM                    20951    638/NetworkManager   
    unix  3      [ ]         STREAM     CONNECTED     22279    707/polkitd          
    unix  3      [ ]         STREAM     CONNECTED     21448    638/NetworkManager   
    unix  3      [ ]         STREAM     CONNECTED     20737    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  2      [ ]         DGRAM                    18676    500/systemd-udevd    
    unix  2      [ ]         DGRAM                    21424    675/crond            
    unix  3      [ ]         STREAM     CONNECTED     18431    490/lvmetad          
    unix  3      [ ]         STREAM     CONNECTED     20787    638/NetworkManager   
    unix  3      [ ]         STREAM     CONNECTED     21129    660/sshd             
    unix  3      [ ]         STREAM     CONNECTED     21449    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  3      [ ]         STREAM     CONNECTED     20269    615/dbus-daemon      
    unix  2      [ ]         DGRAM                    22282    707/polkitd          
    unix  3      [ ]         STREAM     CONNECTED     20965    638/NetworkManager   
    unix  3      [ ]         STREAM     CONNECTED     20321    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  2      [ ]         DGRAM                    23495    858/systemd          
    unix  3      [ ]         STREAM     CONNECTED     20271    472/systemd-journal  /run/systemd/journal/stdout
    unix  2      [ ]         DGRAM                    20274    617/chronyd          
    unix  2      [ ]         DGRAM                    21028    615/dbus-daemon      
    unix  3      [ ]         STREAM     CONNECTED     20736    622/python3          
    unix  2      [ ]         DGRAM                    23487    860/(sd-pam)         
    unix  3      [ ]         DGRAM                    18750    500/systemd-udevd    
    unix  3      [ ]         STREAM     CONNECTED     20610    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  2      [ ]         DGRAM                    18432    1/systemd            
    unix  3      [ ]         STREAM     CONNECTED     20966    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  3      [ ]         DGRAM                    18749    500/systemd-udevd    
    unix  3      [ ]         STREAM     CONNECTED     21398    675/crond            
    unix  3      [ ]         STREAM     CONNECTED     20225    1/systemd            
    unix  3      [ ]         STREAM     CONNECTED     18499    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     20789    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     18664    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     23466    858/systemd          
    unix  3      [ ]         STREAM     CONNECTED     21130    472/systemd-journal  /run/systemd/journal/stdout
    unix  3      [ ]         STREAM     CONNECTED     22280    615/dbus-daemon      /run/dbus/system_bus_socket
    unix  3      [ ]         STREAM     CONNECTED     21399    472/systemd-journal  /run/systemd/journal/stdout
    Active Bluetooth connections (servers and established)
    Proto  Destination       Source            State         PSM DCID   SCID      IMTU    OMTU Security
    Proto  Destination       Source            State     Channel
    
    

    But again by the time I log in, I can immediately start the service.



  • In /etc/sysctl.conf can you add this line?

    net.ipv4.ip_nonlocal_bind = 1
    

    Or just do...

    echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf


  • @scottalanmiller said in HAProxy fails to start on reboot:

    In /etc/sysctl.conf can you add this line?

    net.ipv4.ip_nonlocal_bind = 1
    

    Or just do...

    echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
    

    That did it.
    WTF did I just do? I can read, so I get the general idea.





  • @jaredbusch said in HAProxy fails to start on reboot:

    @scottalanmiller said in HAProxy fails to start on reboot:

    In /etc/sysctl.conf can you add this line?

    net.ipv4.ip_nonlocal_bind = 1
    

    Or just do...

    echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
    

    That did it.
    WTF did I just do? I can read, so I get the general idea.

    For load balancing it needs to be able to bind to non-local ports. For some reason the install did not set this which it seems like it should have done.


Log in to reply