Setting up an ELK Logging Server
-
@scottalanmiller said:
Yeah, they do pre-builts for you. Which is awesome, because ELK is a pain to set up.
I just need to dump my vm and start over haven't had time since last week to put on this project.
-
Okay, got derailed the other day. Working on this more now. DigitalOcean has a guide to setting up LogStash Forwarder on CentOS 7 that I am going through now. Already tried their old CentOS 6 guide, not knowing that there was an updated one, and it didn't work. So seeing if I have better luck with this one. The CentOS 6 guide does not work because OpenSSL has changed.
-
The error that you might encounter is...
2015/03/07 13:08:40.349662 Connecting to [x.x.x.x]:5000 (dny-lnx-log) 2015/03/07 13:08:40.357296 Failed to tls handshake with x.x.x.x x509: certificate is valid for , not dny-lnx-log
Where x.x.x.x is your IP address.
-
@scottalanmiller said:
2015/03/07 13:08:40.349662 Connecting to [x.x.x.x]:5000 (dny-lnx-log)
2015/03/07 13:08:40.357296 Failed to tls handshake with x.x.x.x x509: certificate is valid for , not dny-lnx-logStill does not work after using the CentOS 7 instructions. Am I doing something wrong or is no one else actually testing this? Going to test on Ubuntu.
References for what is actually going wrong...
https://github.com/elasticsearch/logstash-forwarder/issues/221
https://github.com/elasticsearch/logstash-forwarder/issues/221#issuecomment-48390920
http://serverfault.com/questions/633681/logstash-forwarder-is-throwing-ssl-errors -
Tested CentOS 7 on CloudatCost, same issues. No surprise there.
-
Getting a little stuck on this one. No matter what configuration I use, the same error. I think that I am using the right configurations that people show on StackOverflow and GitHub but I keep getting the same errors or nearly the same.
-
Here is my current SSL config file...
# cat ssl.cnf [req] distinguished_name = req_distinguished_name x509_extensions = v3_req prompt = no [req_distinguished_name] C = US ST = NY L = Piffard O = NTG CN = * [v3_req] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer basicConstraints = CA:TRUE subjectAltName = @alt_names [alt_names] DNS.1 = * DNS.2 = *.* DNS.3 = *.*.* DNS.4 = *.*.*.* DNS.5 = *.*.*.*.* DNS.6 = *.*.*.*.*.* DNS.7 = *.*.*.*.*.*.* IP.1 = x.x.x.x IP.2 = 127.0.0.1
-
And here is the current error:
2015/03/07 15:03:34.330398 Connecting to [x.x.x.x]:5000 (x.x.x.x) 2015/03/07 15:03:34.370271 Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs
-
IP and Port are definitely correct. I can test that with a telnet.
-
@scottalanmiller said:
And here is the current error:
2015/03/07 15:03:34.330398 Connecting to [x.x.x.x]:5000 (x.x.x.x) 2015/03/07 15:03:34.370271 Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs
OpenSSL is not my strong point but, this might help http://serverfault.com/questions/611120/failed-tls-handshake-does-not-contain-any-ip-sans
-
Apparently this is no one's strong suit.... OpenSSL made this change and didn't document it. The conversion on GitHub is the actual development team trying to get this sorted out
-
I'm not sure if this is progress or not, but I've gotten it to switch to this error:
2015/03/08 07:40:51.622838 Connecting to [x.x.x.x]:5000 (x.x.x.x) 2015/03/08 07:40:51.659486 Failed to tls handshake with x.x.x.x x509: certificate signed by unknown authority
-
Another guide that I've found. Not sure if it is useful yet.
https://dan.langille.org/2014/08/13/getting-logstash-forwarder-working/
-
Okay, at some point everything got broken and even what was working before is not working now. This is pretty crazy both that all of the instructions from DigitalOcean are wrong, the documentation from ELK is wrong and on GitHub there is no consensus on a fix. I'm rapidly losing all faith in ELK as they are dependent on components that are not working and even they don't seem to have any idea how to make work.
-
Rebuilding from their new (updated in the last 72 hours) ELK image and starting fresh.
-
Update: DigitalOcean has a new build of the ELK image that is fully up to date since I started this thread and you need it in order for things to work. If you are experiencing the issues that I listed above, stop and start over with the latest build. Things "just work" again. I already have CentOS running on CloudatCost sending logs over to ELK on DigitalOcean.
-
If you have a central jump server like we do, it is super easy to push out keys. Once you have the key in place on the Jump server, you can do this to update it at client machines (very easy to script.)
scp /etc/pki/tls/certs/logstash-forwarder.crt root@dny-lnx-pbx1:/etc/pki/tls/certs/
-
Just got a third server feeding into the ELK system. This is working perfectly after the latest update.
-
Here is my working /etc/logstash-forwarder configuration file (x.x.x.x = my IP address, of course)
{ "network": { "servers": [ "x.x.x.x:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt" }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } } ] }
-
Next step is to see if the ElasticSearch YUM repos work for this, because that will be far better than the one off RPM install that DO has us doing in their docs. So let's see.
Here is the docs from ELK.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html