Problems setting up an ELK stack

JaredBusch

Started to setup an ELK stack today. Not a lot of fun. I can not find any good walk through that will setup everything up right as well as educate me on WTF I am doing.

Spun up CentOS7
Installed the custom repos
used yum to install elasticsearch and logstash
followed a couple walkthroughs for kibana.

told of of my Ubiquiti ERL's to send everything to the IP and nothing.
completely empty Kibana screen.

anyone know a good guide?

JaredBusch

Following this: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-logs-on-centos-7

Last night I did it and was seeing the empty Kibana page, but never a security login based on the htpassword setup.

Deleted the VM disk and started over this morning.

Today I am seeing the default Apache page, not the kibana page.

Grr.

JaredBusch

monitoring tail -f /var/log/httpd/error_log

# issue a reload
systemctl reload httpd

results in

[Wed Jan 14 10:45:54.216237 2015] [mpm_prefork:notice] [pid 11168] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Jan 14 10:45:55.272556 2015] [core:notice] [pid 11188] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Jan 14 10:45:55.273544 2015] [suexec:notice] [pid 11188] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jan 14 10:45:55.287436 2015] [auth_digest:notice] [pid 11188] AH01757: generating secret for digest authentication ...
[Wed Jan 14 10:45:55.288409 2015] [lbmethod_heartbeat:notice] [pid 11188] AH02282: No slotmem from mod_heartmonitor
[Wed Jan 14 10:45:55.290883 2015] [mpm_prefork:notice] [pid 11188] AH00163: Apache/2.4.6 (CentOS) configured -- resuming normal operations
[Wed Jan 14 10:45:55.290920 2015] [core:notice] [pid 11188] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

then visiting the page http://elk.domain.com results in

[Wed Jan 14 10:46:02.364936 2015] [autoindex:error] [pid 11190] [client 10.254.103.20:52587] AH01276: Cannot serve directory /var/
www/html/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive

JaredBusch

Apache seems to not even bee loading the vhost config file...

The guide said to create this vhost conf file.

/etc/httpd/conf.d/kibana3.conf

with this info

<VirtualHost elk.domain.com:80>
  ServerName elk.domain.com

  DocumentRoot /var/www/kibana3
  <Directory /var/www/kibana3>
	Allow from all
	Options -Multiviews
  </Directory>

  LogLevel debug
  ErrorLog /var/log/httpd/elk_error_log
  CustomLog /var/log/httpd/elk_access_log combined

  # Set global proxy timeouts
  <Proxy http://127.0.0.1:9200>
	ProxySet connectiontimeout=5 timeout=90
  </Proxy>

  # Proxy for _aliases and .*/_search
  <LocationMatch "^/(_nodes|_aliases|.*/_aliases|_search|.*/_search|_mapping|.*/_mapping)$">
	ProxyPassMatch http://127.0.0.1:9200/$1
	ProxyPassReverse http://127.0.0.1:9200/$1
  </LocationMatch>

  # Proxy for kibana-int/{dashboard,temp} stuff (if you don't want auth on /, then you will want these to be protected)
  <LocationMatch "^/(kibana-int/dashboard/|kibana-int/temp)(.*)$">
	ProxyPassMatch http://127.0.0.1:9200/$1$2
	ProxyPassReverse http://127.0.0.1:9200/$1$2
  </LocationMatch>

  <Location />
	AuthType Basic
	AuthBasicProvider file
	AuthName "Restricted"
	AuthUserFile /etc/httpd/conf.d/kibana-htpasswd
	Require valid-user
  </Location>
</VirtualHost>

JaredBusch

My notes so far. to create this non-working system.

https://jaredbusch.com/2015/01/14/setup-an-elk-server-to-handle-system-logs/

gjacobse

Sorry to hear of the difficulty setting this up. But as I am interested in doing the same,.. maybe I'll skip the issues.