SysLog Forwarding for XenServer

DustinB3403

I'll run it again though.

DustinB3403

So still digging into this...

[root@syslog-cent bin]# ./kibana serve restart
  log   [10:14:12.914] [fatal] Error: listen EADDRINUSE 0.0.0.0:5601
	at Object.exports._errnoException (util.js:870:11)
	at exports._exceptionWithHostPort (util.js:893:20)
	at Server._listen2 (net.js:1236:14)
	at listen (net.js:1272:10)
	at net.js:1381:9
	at nextTickCallbackWith3Args (node.js:448:9)
	at process._tickDomainCallback (node.js:395:17)
FATAL { [Error: listen EADDRINUSE 0.0.0.0:5601]
  cause:
   { [Error: listen EADDRINUSE 0.0.0.0:5601]
	 code: 'EADDRINUSE',
	 errno: 'EADDRINUSE',
	 syscall: 'listen',
	 address: '0.0.0.0',
	 port: 5601 },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen',
  address: '0.0.0.0',
  port: 5601 }

DustinB3403

We must have to change the kibana.yml file to not listen on the localhost address...

kibana.yml...

[root@syslog-cent config]# cat kibana.yml
server.host: "localhost"
elasticsearch_url: "http://localhost:9200"
server.port:5601

DustinB3403

I'm rebooting see if its hung somewhere. As from what I can find online the kibana server is supposedly running twice...

DustinB3403

Ok so after playing with the timestamp (top right) I do actually have logs, but only from the 12th of the month...

So maybe it was working, but not showing the logs... now to figure out what the crap is broken....

momurda

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool. Just set Xencenter to forward logs to the Graylog server, seems to work well. Xenserver still making local log entries, but i am ok with that.
Xenserver sure does like logging messages. 2 hosts making a couple hundred messages/minute, xenstored and xapi are the top ones by far.

DustinB3403

@momurda said in SysLog Forwarding for XenServer:

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool. Just set Xencenter to forward logs to the Graylog server, seems to work well. Xenserver still making local log entries, but i am ok with that.
Xenserver sure does like logging messages. 2 hosts making a couple hundred messages/minute, xenstored and xapi are the top ones by far.

What source are you using?

momurda

All i did was download and import the ova, then went into Xencenter and forwarded logs on each host to the ip of the graylog server. Here is my sources page
0_1471548705190_upload-f2c491d5-e547-43ae-bbe7-1cfaeb259539

Here is more sources, basically the whole list. I am still quite overwhelmed with the options and config of graylog, but as i get dashboards setup for things and add more log sources i will post them here as well if you would like.

0_1471548847262_upload-5fae783d-0560-4963-b4af-b4000cdc21b3

momurda

I think it is important to note that the graylog ova is preconfigured to 'just work' according to their site, and it seems to do just that. I will try adding some of my windows vm to this and see what happens later today or tomorrow.

DustinB3403

For some reason I thought / think there are some pretty big limitations to GrayLog.

Maybe I'm wrong.... but I'll take a look at it.

DustinB3403

For anyone curious how to stop any local logging just modify

/var/lib/syslog.conf

Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.

BRRABill

@DustinB3403 said in SysLog Forwarding for XenServer:

For anyone curious how to stop any local logging just modify
/var/lib/syslog.conf
Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.

Reboot and see if it sticks.

It did not for me.

BRRABill

@momurda said in SysLog Forwarding for XenServer:

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool.

I cannot get it to import onto my XS.

Did you just import it in with no issues?

DustinB3403

@BRRABill said in SysLog Forwarding for XenServer:

@DustinB3403 said in SysLog Forwarding for XenServer:
For anyone curious how to stop any local logging just modify
/var/lib/syslog.conf
Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.
Reboot and see if it sticks.

It did not for me.

Will test tomorrow.

BRRABill

@DustinB3403 said in SysLog Forwarding for XenServer:

@BRRABill said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
For anyone curious how to stop any local logging just modify
/var/lib/syslog.conf
Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.
Reboot and see if it sticks.

It did not for me.
Will test tomorrow.

That was my issue. On reboot it would wipe out the changes I made.

stacksofplates

@BRRABill said in SysLog Forwarding for XenServer:

@DustinB3403 said in SysLog Forwarding for XenServer:
@BRRABill said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
For anyone curious how to stop any local logging just modify
/var/lib/syslog.conf
Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.
Reboot and see if it sticks.

It did not for me.
Will test tomorrow.
That was my issue. On reboot it would wipe out the changes I made.

In a pinch you can do chattr +i on the rsyslog.conf file to make it immutable.

BRRABill

@stacksofplates said in SysLog Forwarding for XenServer:

@BRRABill said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
@BRRABill said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
For anyone curious how to stop any local logging just modify
/var/lib/syslog.conf
Comment out everything that hits a local path, leaving the @<ip_addr> as the only option.
Reboot and see if it sticks.

It did not for me.
Will test tomorrow.
That was my issue. On reboot it would wipe out the changes I made.
In a pinch you can do chattr +i on the rsyslog.conf file to make it immutable.

Yeah on the bottom of that article everyone talks about it basically says to change the permission to make it unwritable.

But they call that a QUOTE dirty, dirty tirck UNQUOTE.

BRRABill

@BRRABill said in SysLog Forwarding for XenServer:

@momurda said in SysLog Forwarding for XenServer:

I donwloaded the Graylog OVA this morning to test it out and put it on my XS pool.

I cannot get it to import onto my XS.

Did you just import it in with no issues?

I tried on my other XS and it worked fine.

Must have been a memory issue on my test one.

BRRABill

@momurda said in SysLog Forwarding for XenServer:

I think it is important to note that the graylog ova is preconfigured to 'just work' according to their site, and it seems to do just that. I will try adding some of my windows vm to this and see what happens later today or tomorrow.

Once I had it imported onto my XS, I had it logging in seconds.

Pretty sweet.

If only I was using open source and virtualization years ago!!!!!!

momurda

@BRRABill
Yes, no issues. Didn't even use the fixup disc option. Took a few minutes to start up but it worked right away