Setting up an ELK Logging Server

scottalanmiller

Today's project for me is building an ELK logging solution. ELK is a stack name. The ELK stack is comprised of ElasticSearch, LogStash and Kibana and is designed to be a direct competitor to Splunk. ELK is completely open source and free.

Digital Ocean has a simple tutorial for getting started with ELK on DO.

Set up the Droplet, added my Jump Server SSH key, fired it up and Kibana is working great.

The DigitalOcean guide suggests the smallest system to consider is the dual core, 2GB system. But so far we are running well on the single core, 1GB droplet.

I'll be trying this later for sure. Digital Ocean has tons of documentation available, it's really great!

coliver

I'm going to try this out. Thanks for bringing it up. Going to try it for my hyper-v environment.

MattSpeller

Just be careful it doesn't log logs of your logs :trollface:

JaredBusch

I am going to have to read this tutorial tonight because I tried to setup an ELK server on an internal VM at a client last week and could not get it going.

Was just a CentOS 7 minimal install and went from there.

JaredBusch

Bah, useless as I expected because it is all pre-built into the droplet.

scottalanmiller

@JaredBusch said:

Bah, useless as I expected because it is all pre-built into the droplet.

Yeah, they do pre-builts for you. Which is awesome, because ELK is a pain to set up.

JaredBusch

@scottalanmiller said:

Yeah, they do pre-builts for you. Which is awesome, because ELK is a pain to set up.

I just need to dump my vm and start over haven't had time since last week to put on this project.

scottalanmiller

Okay, got derailed the other day. Working on this more now. DigitalOcean has a guide to setting up LogStash Forwarder on CentOS 7 that I am going through now. Already tried their old CentOS 6 guide, not knowing that there was an updated one, and it didn't work. So seeing if I have better luck with this one. The CentOS 6 guide does not work because OpenSSL has changed.

scottalanmiller

The error that you might encounter is...

2015/03/07 13:08:40.349662 Connecting to [x.x.x.x]:5000 (dny-lnx-log)
2015/03/07 13:08:40.357296 Failed to tls handshake with x.x.x.x x509: certificate is valid for , not dny-lnx-log

Where x.x.x.x is your IP address.

scottalanmiller

@scottalanmiller said:

2015/03/07 13:08:40.349662 Connecting to [x.x.x.x]:5000 (dny-lnx-log)
2015/03/07 13:08:40.357296 Failed to tls handshake with x.x.x.x x509: certificate is valid for , not dny-lnx-log

Still does not work after using the CentOS 7 instructions. Am I doing something wrong or is no one else actually testing this? Going to test on Ubuntu.

References for what is actually going wrong...

https://github.com/elasticsearch/logstash-forwarder/issues/221
https://github.com/elasticsearch/logstash-forwarder/issues/221#issuecomment-48390920
http://serverfault.com/questions/633681/logstash-forwarder-is-throwing-ssl-errors

scottalanmiller

Tested CentOS 7 on CloudatCost, same issues. No surprise there.

scottalanmiller

Getting a little stuck on this one. No matter what configuration I use, the same error. I think that I am using the right configurations that people show on StackOverflow and GitHub but I keep getting the same errors or nearly the same.

scottalanmiller

Here is my current SSL config file...

# cat ssl.cnf
[req] 
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = NY
L =  Piffard
O = NTG
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = *
DNS.2 = *.*
DNS.3 = *.*.*
DNS.4 = *.*.*.*
DNS.5 = *.*.*.*.*
DNS.6 = *.*.*.*.*.*
DNS.7 = *.*.*.*.*.*.*
IP.1 = x.x.x.x
IP.2 = 127.0.0.1

scottalanmiller

And here is the current error:

2015/03/07 15:03:34.330398 Connecting to [x.x.x.x]:5000 (x.x.x.x)
2015/03/07 15:03:34.370271 Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

scottalanmiller

IP and Port are definitely correct. I can test that with a telnet.

@scottalanmiller said:

And here is the current error:

2015/03/07 15:03:34.330398 Connecting to [x.x.x.x]:5000 (x.x.x.x)
2015/03/07 15:03:34.370271 Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

OpenSSL is not my strong point but, this might help http://serverfault.com/questions/611120/failed-tls-handshake-does-not-contain-any-ip-sans

scottalanmiller

Apparently this is no one's strong suit.... OpenSSL made this change and didn't document it. The conversion on GitHub is the actual development team trying to get this sorted out

scottalanmiller

I'm not sure if this is progress or not, but I've gotten it to switch to this error:

2015/03/08 07:40:51.622838 Connecting to [x.x.x.x]:5000 (x.x.x.x)
2015/03/08 07:40:51.659486 Failed to tls handshake with x.x.x.x x509: certificate signed by unknown authority

scottalanmiller

Another guide that I've found. Not sure if it is useful yet.

https://dan.langille.org/2014/08/13/getting-logstash-forwarder-working/