SysLog Forwarding for XenServer

DustinB3403

Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 32ms - 9.0B"}
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 8ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":38,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 38ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":24,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 24ms - 9.0B"}
Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 15ms - 9.0B"}
Aug 16 08:41:16 syslog-cent systemd: Starting Cleanup of Temporary Directories...
Aug 16 08:41:16 syslog-cent systemd: Started Cleanup of Temporary Directories.

DustinB3403

I don't see any error messages in the above logs.

So what did I mess up?

DustinB3403

In /var/log/kibana/kibana.stout I have the below...

{"type":"log","@timestamp":"2016-08-15T15:43:07+00:00","tags":["fatal"],"pid":23942,"level":"fatal","message":"listen EADDRINUSE 127.0.0.1:5601","error":{"message":"listen EADDRINUSE 127.0.0.1:5601","name":"Error","stack":"Error: listen EADDRINUSE 127.0.0.1:5601\n    at Object.exports._errnoException (util.js:870:11)\n    at exports._exceptionWithHostPort (util.js:893:20)\n    at Server._listen2 (net.js:1236:14)\n    at listen (net.js:1272:10)\n    at net.js:1381:9\n    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:63:16)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:82:10)","code":"EADDRINUSE"}}

And in kibana.stderr

[root@syslog-cent kibana]# tail kibana.stderr
	 errno: 'EADDRINUSE',
	 syscall: 'listen',
	 address: '127.0.0.1',
	 port: 5601 },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen',
  address: '127.0.0.1',
  port: 5601 }

Is the system listening to the wrong port? Shouldn't it be 514 or 5140?

DustinB3403

So in checking out the firewall on the Kibana server using nmap...

Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-16 09:34 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000089s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
9200/tcp open  wap-wsp
No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=6.40%E=4%D=8/16%OT=22%CT=1%CU=32095%PV=N%DS=0%DC=L%G=Y%TM=57B3166
OS:E%P=x86_64-redhat-linux-gnu)SEQ(SP=104%GCD=1%ISR=10A%TI=Z%CI=I%II=I%TS=A
OS:)OPS(O1=MFFD7ST11NW7%O2=MFFD7ST11NW7%O3=MFFD7NNT11NW7%O4=MFFD7ST11NW7%O5
OS:=MFFD7ST11NW7%O6=MFFD7ST11)WIN(W1=AAAA%W2=AAAA%W3=AAAA%W4=AAAA%W5=AAAA%W
OS:6=AAAA)ECN(R=Y%DF=Y%T=40%W=AAAA%O=MFFD7NNSNW7%CC=Y%Q=)T1(R=Y%DF=Y%T=40%S
OS:=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%R
OS:D=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=40%W=
OS:0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)U
OS:1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DF
OS:I=N%T=40%CD=S)

Network Distance: 0 hops

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.68 seconds

travisdh1

@DustinB3403 You'd need at least 5601 open, right? What's the output of

fireall-cmd --list-all

DustinB3403

@travisdh1

[root@syslog-cent log]# firewall-cmd --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 80/tcp 5044/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

travisdh1

We may be getting somewhere. You're kibana.stderr looks like you need port 5601 open.

firewall-cmd --zone=public --add-port=5601/tcp --permanent

Sidenote: I still don't like firewall-cmd. Change is hard, even for geeks.

DustinB3403

@travisdh1 So with both TCP and UDP open.

[root@syslog-cent log]# firewall-cmd --list-all
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 5601/udp 80/tcp 5601/tcp 5044/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Still nothing showing up in Kibana

coliver

@travisdh1 said in SysLog Forwarding for XenServer:

We may be getting somewhere. You're kibana.stderr looks like you need port 5601 open.
firewall-cmd --zone=public --add-port=5601/tcp --permanent
Sidenote: I still don't like firewall-cmd. Change is hard, even for geeks.

Do you have to reload the firewalld settings to get them to apply?

firewall-cmd --reload

travisdh1

@coliver Yep, I keep forgetting that step.... spent an hour the other day wondering why things weren't working.

DustinB3403

@coliver I did.

I'll run it again though.

DustinB3403

So still digging into this...

[root@syslog-cent bin]# ./kibana serve restart
  log   [10:14:12.914] [fatal] Error: listen EADDRINUSE 0.0.0.0:5601
	at Object.exports._errnoException (util.js:870:11)
	at exports._exceptionWithHostPort (util.js:893:20)
	at Server._listen2 (net.js:1236:14)
	at listen (net.js:1272:10)
	at net.js:1381:9
	at nextTickCallbackWith3Args (node.js:448:9)
	at process._tickDomainCallback (node.js:395:17)
FATAL { [Error: listen EADDRINUSE 0.0.0.0:5601]
  cause:
   { [Error: listen EADDRINUSE 0.0.0.0:5601]
	 code: 'EADDRINUSE',
	 errno: 'EADDRINUSE',
	 syscall: 'listen',
	 address: '0.0.0.0',
	 port: 5601 },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen',
  address: '0.0.0.0',
  port: 5601 }

DustinB3403

We must have to change the kibana.yml file to not listen on the localhost address...

kibana.yml...

[root@syslog-cent config]# cat kibana.yml
server.host: "localhost"
elasticsearch_url: "http://localhost:9200"
server.port:5601

DustinB3403

I'm rebooting see if its hung somewhere. As from what I can find online the kibana server is supposedly running twice...

DustinB3403

Ok so after playing with the timestamp (top right) I do actually have logs, but only from the 12th of the month...

So maybe it was working, but not showing the logs... now to figure out what the crap is broken....

momurda

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool. Just set Xencenter to forward logs to the Graylog server, seems to work well. Xenserver still making local log entries, but i am ok with that.
Xenserver sure does like logging messages. 2 hosts making a couple hundred messages/minute, xenstored and xapi are the top ones by far.

DustinB3403

@momurda said in SysLog Forwarding for XenServer:

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool. Just set Xencenter to forward logs to the Graylog server, seems to work well. Xenserver still making local log entries, but i am ok with that.
Xenserver sure does like logging messages. 2 hosts making a couple hundred messages/minute, xenstored and xapi are the top ones by far.

What source are you using?

momurda

All i did was download and import the ova, then went into Xencenter and forwarded logs on each host to the ip of the graylog server. Here is my sources page
0_1471548705190_upload-f2c491d5-e547-43ae-bbe7-1cfaeb259539

Here is more sources, basically the whole list. I am still quite overwhelmed with the options and config of graylog, but as i get dashboards setup for things and add more log sources i will post them here as well if you would like.

0_1471548847262_upload-5fae783d-0560-4963-b4af-b4000cdc21b3

momurda

I think it is important to note that the graylog ova is preconfigured to 'just work' according to their site, and it seems to do just that. I will try adding some of my windows vm to this and see what happens later today or tomorrow.

DustinB3403

For some reason I thought / think there are some pretty big limitations to GrayLog.

Maybe I'm wrong.... but I'll take a look at it.