SysLog Forwarding for XenServer

DustinB3403

Since I'm having a hell of time getting this going, I setup a KiwI Syslog on a VM from one of my host, and it just works.

Just enabling the logging to the IP address, and let it go.

scottalanmiller

@DustinB3403 said in SysLog Forwarding for XenServer:

Since I'm having a hell of time getting this going, I setup a KiwI Syslog on a VM from one of my host, and it just works.

Just enabling the logging to the IP address, and let it go.

Instead of posting the logs to diagnose?

DustinB3403

@scottalanmiller Again, where do I look for them....

BRRABill

@DustinB3403 said in SysLog Forwarding for XenServer:

@scottalanmiller Again, where do I look for them....

Check the logs.

(Boy I am glad I am not withing physical reach of you! )

DustinB3403

My point with the Kiwi server is that I must've misconfigured something on the SysLog installation.

Since I've made no direct change other than the logging address in the XC settings. (which does update the settings in the server) but it doesn't change the default port etc.

scottalanmiller

@DustinB3403 said in SysLog Forwarding for XenServer:

@scottalanmiller Again, where do I look for them....

/var/log/messages same as always

DustinB3403

@scottalanmiller said in SysLog Forwarding for XenServer:

@DustinB3403 said in SysLog Forwarding for XenServer:

@scottalanmiller Again, where do I look for them....

/var/log/messages same as always

... and what would be a decent way to view this as it breezes by at 100 lines a second?

Is there a specific event you're looking for?

scottalanmiller

@DustinB3403 said in SysLog Forwarding for XenServer:

@scottalanmiller said in SysLog Forwarding for XenServer:

@DustinB3403 said in SysLog Forwarding for XenServer:

@scottalanmiller Again, where do I look for them....

/var/log/messages same as always

... and what would be a decent way to view this as it breezes by at 100 lines a second?

Is there a specific event you're looking for?

It only breezes by if you tail it. Try just looking at it statically.

What is generating so many messages?

scottalanmiller

Looking for errors from the forwarder.

dafyre

Have you tried? If you're seeing logs coming in from XenServer, then you should be on the right track.

tail -f|grep nameofsourceserver

DustinB3403

So this is what I have currently with the Kibana system running.

@dafyre tail -f|grep servername results in "tail: warning: following standard input indefinitely is ineffective"

DustinB3403

Here it is with me connected to the system, and my server supposedly sending logs to it.

[root@syslog-cent ~]# tail /var/log/messages
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho                                                                                                     st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":                                                                                                     "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co                                                                                                     ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan                                                                                                     guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat                                                                                                     usCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B"                                                                                                     }
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho                                                                                                     st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":                                                                                                     "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co                                                                                                     ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan                                                                                                     guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat                                                                                                     usCode":200,"responseTime":29,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 29ms - 9.0B"                                                                                                     }
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host"                                                                                                     :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.                                                                                                     4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte                                                                                                     nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua                                                                                                     ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC                                                                                                     ode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host"                                                                                                     :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.                                                                                                     4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte                                                                                                     nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua                                                                                                     ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC                                                                                                     ode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 32ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers"                                                                                                     :{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://1                                                                                                     92.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.                                                                                                     0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding                                                                                                     ":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.                                                                                                     83/app/kibana?"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_un                                                                                                     available=true&preference=1471347138543 200 8ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho                                                                                                     st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":                                                                                                     "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co                                                                                                     ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan                                                                                                     guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat                                                                                                     usCode":200,"responseTime":38,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 38ms - 9.0B"                                                                                                     }
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host"                                                                                                     :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.                                                                                                     4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte                                                                                                     nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua                                                                                                     ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC                                                                                                     ode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho                                                                                                     st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":                                                                                                     "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co                                                                                                     ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan                                                                                                     guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat                                                                                                     usCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B"                                                                                                     }
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host"                                                                                                     :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.                                                                                                     4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte                                                                                                     nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua                                                                                                     ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC                                                                                                     ode":200,"responseTime":24,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 24ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta                                                                                                     tusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers"                                                                                                     :{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://1                                                                                                     92.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.                                                                                                     0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding                                                                                                     ":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.                                                                                                     83/app/kibana?"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_u                                                                                                     navailable=true&preference=1471347138543 200 15ms - 9.0B"}

DustinB3403

Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 32ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 8ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":38,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 38ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":24,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 24ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 15ms - 9.0B"}
Aug 16 08:41:16 syslog-cent systemd: Starting Cleanup of Temporary Directories...
Aug 16 08:41:16 syslog-cent systemd: Started Cleanup of Temporary Directories.

DustinB3403

I don't see any error messages in the above logs.

So what did I mess up?

DustinB3403

In /var/log/kibana/kibana.stout I have the below...

{"type":"log","@timestamp":"2016-08-15T15:43:07+00:00","tags":["fatal"],"pid":23942,"level":"fatal","message":"listen EADDRINUSE 127.0.0.1:5601","error":{"message":"listen EADDRINUSE 127.0.0.1:5601","name":"Error","stack":"Error: listen EADDRINUSE 127.0.0.1:5601\n    at Object.exports._errnoException (util.js:870:11)\n    at exports._exceptionWithHostPort (util.js:893:20)\n    at Server._listen2 (net.js:1236:14)\n    at listen (net.js:1272:10)\n    at net.js:1381:9\n    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:63:16)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:82:10)","code":"EADDRINUSE"}}

And in kibana.stderr

[root@syslog-cent kibana]# tail kibana.stderr
	 errno: 'EADDRINUSE',
	 syscall: 'listen',
	 address: '127.0.0.1',
	 port: 5601 },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen',
  address: '127.0.0.1',
  port: 5601 }

Is the system listening to the wrong port? Shouldn't it be 514 or 5140?

DustinB3403

So in checking out the firewall on the Kibana server using nmap...

Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-16 09:34 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000089s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
9200/tcp open  wap-wsp
No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=6.40%E=4%D=8/16%OT=22%CT=1%CU=32095%PV=N%DS=0%DC=L%G=Y%TM=57B3166
OS:E%P=x86_64-redhat-linux-gnu)SEQ(SP=104%GCD=1%ISR=10A%TI=Z%CI=I%II=I%TS=A
OS:)OPS(O1=MFFD7ST11NW7%O2=MFFD7ST11NW7%O3=MFFD7NNT11NW7%O4=MFFD7ST11NW7%O5
OS:=MFFD7ST11NW7%O6=MFFD7ST11)WIN(W1=AAAA%W2=AAAA%W3=AAAA%W4=AAAA%W5=AAAA%W
OS:6=AAAA)ECN(R=Y%DF=Y%T=40%W=AAAA%O=MFFD7NNSNW7%CC=Y%Q=)T1(R=Y%DF=Y%T=40%S
OS:=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%R
OS:D=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=40%W=
OS:0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)U
OS:1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DF
OS:I=N%T=40%CD=S)

Network Distance: 0 hops

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.68 seconds

travisdh1

@DustinB3403 You'd need at least 5601 open, right? What's the output of

fireall-cmd --list-all

DustinB3403

@travisdh1

[root@syslog-cent log]# firewall-cmd --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 80/tcp 5044/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

travisdh1

We may be getting somewhere. You're kibana.stderr looks like you need port 5601 open.

firewall-cmd --zone=public --add-port=5601/tcp --permanent

Sidenote: I still don't like firewall-cmd. Change is hard, even for geeks.

DustinB3403

@travisdh1 So with both TCP and UDP open.

[root@syslog-cent log]# firewall-cmd --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 5601/udp 80/tcp 5601/tcp 5044/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Still nothing showing up in Kibana