What is the difference between Dead and Failed for service status?



  • I am doing some testing to monitor services via command. I need an expected response when a service is considered down.

    During my testing I got two different results after stopping a service.

    ################

    First service - Suricata

    #################

    sudo systemctl show -p SubState --value suricata

    the state returned from that command is running as we would expect.

    33a7a301-df49-436b-9e13-912ba1938ae2-image.png

    Now we run sudo systemctl stop suricata

    Then run sudo systemctl show -p SubState --value suricata which returns failed.

    c69319b0-5737-4475-a30c-cf3d4928603c-image.png

    ################

    Second service - Wazuh-Agent

    #################

    sudo systemctl show -p SubState --value wazuh-agent

    The state returned from that command is running as we would expect.

    c83c0d4b-b27f-4d54-bd10-14ee2fffb8ec-image.png

    Now we run sudo systemctl stop wazuh-agent

    Then run sudo systemctl show -p SubState --value wazuh-agent which returns dead.

    e086a825-f771-44f8-81b2-de34d197ffa3-image.png



  • Did you look at man systemctl to get more details yet? I just checked, and while I don't see the specific status listed in the manual, there are a few links to additional resources.



  • From the docs I can find, the service isn't set to start on boot, and also isn't running. So it would be inactive (dead) in that case.



  • It looks like dead is the default state of all other services after being stopped.



  • But this is a child process of Wazuh, no? So would the forking flag need to be set to accurately report the status of the service?

    Not about wazuh, but may pertain.



  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    From the docs I can find, the service isn't set to start on boot, and also isn't running. So it would be inactive (dead) in that case.

    Yeah the parameter ActiveState would return inactive

    SubState returns dead



  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    But this is a child process of Wazuh, no? So would the forking flag need to be set to accurately report the status of the service?

    Not about wazuh, but may pertain.

    No these ones are the children

    c317457d-4aff-4671-a39e-37e4609822ee-image.png



  • @IRJ said in What is the difference between Dead and Failed for service status?:

    sudo systemctl show -p SubState --value wazuh-agent

    Oh I see what you have going on (a bit tired cranky kid last night).

    From the top

    Active: active (running) means the service is setup to start on boot and is running.

    Active: failed means the service crashed for some reason. Check journalctl for more details

    Active: inactive (dead) means not set to start on boot, and not running.



  • Also you could setup your monitoring to report/alert for "Anything but active (running)". It would have the same effect, because in any case of the services not running this system function stops working.



  • Did you go through the systemctl enable wazuh-agent so the service is started at boot?



  • There are also other SUB status as show here

     systemctl list-units --type service --all
      UNIT                                                  LOAD      ACTIVE   SUB     DESCRIPTION
      auditd.service                                        loaded    active   running Security Auditing Service
      brandbot.service                                      loaded    inactive dead    Flexible Branding Service
      chronyd.service                                       loaded    active   running NTP client/server
      cpupower.service                                      loaded    inactive dead    Configure CPU power related settings
      crond.service                                         loaded    active   running Command Scheduler
      dbus.service                                          loaded    active   running D-Bus System Message Bus
    ● display-manager.service                               not-found inactive dead    display-manager.service
      dm-event.service                                      loaded    inactive dead    Device-mapper event daemon
      dracut-shutdown.service                               loaded    inactive dead    Restore /run/initramfs
      ebtables.service                                      loaded    inactive dead    Ethernet Bridge Filtering tables
      emergency.service                                     loaded    inactive dead    Emergency Shell
    ● exim.service                                          not-found inactive dead    exim.service
      firewalld.service                                     loaded    active   running firewalld - dynamic firewall daemon
      [email protected]                                    loaded    active   running Getty on tty1
      httpd.service                                         loaded    active   running The Apache HTTP Server
    ● ip6tables.service                                     not-found inactive dead    ip6tables.service
    ● ipset.service                                         not-found inactive dead    ipset.service
    ● iptables.service                                      not-found inactive dead    iptables.service
      irqbalance.service                                    loaded    inactive dead    irqbalance daemon
      kdump.service                                         loaded    active   exited  Crash recovery kernel arming
      kmod-static-nodes.service                             loaded    active   exited  Create list of required static device nodes for the current kernel
    ● lvm2-activation.service                               not-found inactive dead    lvm2-activation.service
      lvm2-lvmetad.service                                  loaded    active   running LVM2 metadata daemon
      lvm2-lvmpolld.service                                 loaded    inactive dead    LVM2 poll daemon
      lvm2-monitor.service                                  loaded    active   exited  Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
      [email protected]:2.service                               loaded    active   exited  LVM2 PV scan on device 8:2
      mariadb.service                                       loaded    active   running MariaDB database server
      microcode.service                                     loaded    inactive dead    Load CPU microcode update
      network.service                                       loaded    active   exited  LSB: Bring up/down networking
      NetworkManager-wait-online.service                    loaded    active   exited  Network Manager Wait Online
      NetworkManager.service                                loaded    active   running Network Manager
    ● ntpd.service                                          not-found inactive dead    ntpd.service
    ● ntpdate.service                                       not-found inactive dead    ntpdate.service
      plymouth-quit-wait.service                            loaded    inactive dead    Wait for Plymouth Boot Screen to Quit
      plymouth-quit.service                                 loaded    inactive dead    Terminate Plymouth Boot Screen
      plymouth-read-write.service                           loaded    inactive dead    Tell Plymouth To Write Out Runtime Data
      plymouth-start.service                                loaded    inactive dead    Show Plymouth Boot Screen
      polkit.service                                        loaded    active   running Authorization Manager
      postfix.service                                       loaded    active   running Postfix Mail Transport Agent
      rc-local.service                                      loaded    inactive dead    /etc/rc.d/rc.local Compatibility
      rescue.service                                        loaded    inactive dead    Rescue Shell
      rhel-autorelabel-mark.service                         loaded    inactive dead    Mark the need to relabel after reboot
      rhel-autorelabel.service                              loaded    inactive dead    Relabel all filesystems, if necessary
      rhel-configure.service                                loaded    inactive dead    Reconfigure the system on administrator request
      rhel-dmesg.service                                    loaded    active   exited  Dump dmesg to /var/log/dmesg
      rhel-domainname.service                               loaded    active   exited  Read and set NIS domainname from /etc/sysconfig/network
      rhel-import-state.service                             loaded    active   exited  Import network configuration from initramfs


  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    Did you go through the systemctl enable wazuh-agent so the service is started at boot?

    Yeah that has been done. I was manually stopping services to force changing state so I could see the output.



  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    Also you could setup your monitoring to report/alert for "Anything but active (running)". It would have the same effect, because in any case of the services not running this system function stops working.

    I found another anomaly

    e2d42315-3c20-4242-85b3-a150e0d59fa2-image.png

    I get this when I check app armor status via apparmor_status

    96d53d2f-8edb-4d0a-af9f-3a6e30a04db7-image.png



  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    There are also other SUB status as show here

    systemctl list-units --type service --all
    UNIT LOAD ACTIVE SUB DESCRIPTION
    auditd.service loaded active running Security Auditing Service
    brandbot.service loaded inactive dead Flexible Branding Service
    chronyd.service loaded active running NTP client/server
    cpupower.service loaded inactive dead Configure CPU power related settings
    crond.service loaded active running Command Scheduler
    dbus.service loaded active running D-Bus System Message Bus
    ● display-manager.service not-found inactive dead display-manager.service
    dm-event.service loaded inactive dead Device-mapper event daemon
    dracut-shutdown.service loaded inactive dead Restore /run/initramfs
    ebtables.service loaded inactive dead Ethernet Bridge Filtering tables
    emergency.service loaded inactive dead Emergency Shell
    ● exim.service not-found inactive dead exim.service
    firewalld.service loaded active running firewalld - dynamic firewall daemon
    [email protected] loaded active running Getty on tty1
    httpd.service loaded active running The Apache HTTP Server
    ● ip6tables.service not-found inactive dead ip6tables.service
    ● ipset.service not-found inactive dead ipset.service
    ● iptables.service not-found inactive dead iptables.service
    irqbalance.service loaded inactive dead irqbalance daemon
    kdump.service loaded active exited Crash recovery kernel arming
    kmod-static-nodes.service loaded active exited Create list of required static device nodes for the current kernel
    ● lvm2-activation.service not-found inactive dead lvm2-activation.service
    lvm2-lvmetad.service loaded active running LVM2 metadata daemon
    lvm2-lvmpolld.service loaded inactive dead LVM2 poll daemon
    lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
    [email protected]:2.service loaded active exited LVM2 PV scan on device 8:2
    mariadb.service loaded active running MariaDB database server
    microcode.service loaded inactive dead Load CPU microcode update
    network.service loaded active exited LSB: Bring up/down networking
    NetworkManager-wait-online.service loaded active exited Network Manager Wait Online
    NetworkManager.service loaded active running Network Manager
    ● ntpd.service not-found inactive dead ntpd.service
    ● ntpdate.service not-found inactive dead ntpdate.service
    plymouth-quit-wait.service loaded inactive dead Wait for Plymouth Boot Screen to Quit
    plymouth-quit.service loaded inactive dead Terminate Plymouth Boot Screen
    plymouth-read-write.service loaded inactive dead Tell Plymouth To Write Out Runtime Data
    plymouth-start.service loaded inactive dead Show Plymouth Boot Screen
    polkit.service loaded active running Authorization Manager
    postfix.service loaded active running Postfix Mail Transport Agent
    rc-local.service loaded inactive dead /etc/rc.d/rc.local Compatibility
    rescue.service loaded inactive dead Rescue Shell
    rhel-autorelabel-mark.service loaded inactive dead Mark the need to relabel after reboot
    rhel-autorelabel.service loaded inactive dead Relabel all filesystems, if necessary
    rhel-configure.service loaded inactive dead Reconfigure the system on administrator request
    rhel-dmesg.service loaded active exited Dump dmesg to /var/log/dmesg
    rhel-domainname.service loaded active exited Read and set NIS domainname from /etc/sysconfig/network
    rhel-import-state.service loaded active exited Import network configuration from initramfs

    Probably best to just look at active and inactive.



  • I tried using FirstState and still receiving failed from suricata.

    ac4c1efa-8518-4125-88b5-c28d0c33fb46-image.png `



  • @IRJ Did you look at journalctl to see what the logs say?



  • @DustinB3403 said in What is the difference between Dead and Failed for service status?:

    @IRJ Did you look at journalctl to see what the logs say?

    He doesn't care about why things are failed. He just wants the states. It is for monitoring and automation.