@scottalanmiller Hm... I'd thought of posting that up here but figured it'd be old news for you lassi it hounds... Good news for the battle for ROSS.

@coliver said in Log & Alerts Management:

Graylog would be the solution for that.

Recognise that name will have to look into that again

That will return from the default index. To specify an index add it in the URL:

curl 192.168.1.100:9200/index/_search?pretty'

Tags added.

https://github.com/ElasticHQ/elasticsearch-HQ

So I don't believe it handles snapshots yet, but it still looks like a pretty useful tool that I have been meaning to try out for people who don't want to manage through the API.

hi @mhamed, if you are solved this step i need your help because I'm currently working on same Project .

Got it. The node list needs to be master nodes only, but by default the non-master local 127.0.0.1 is left in the list. You have to remove it but keep the other nodes in for it to work.

@dafyre said in SysLog Forwarding for XenServer:

@BRRABill said in SysLog Forwarding for XenServer:

I am the new King of Open Source.

H aha ha. How's that?

It's my answer to anything.

Need a new logging server? Open Source!

Need a new XXXXXX? Open Source!

P.S.; While the ability to "pivot" from e.g. alert to metrics to log seamlessly from w/in a single UI is indeed attractive, the time series data model of the PLG stack (Prometheus Loki Grafana) does not lend itself well to "The Tail at Scale" problem.

https://www2.cs.duke.edu/courses/cps296.4/fall13/838-CloudPapers/dean_longtail.pdf

IOW; it is all a lot more complex than one may initially imagine... lol.

Graylog has updated and no longer relies on the old version of ElasticSearch. It will use ElasticSearch 2 now. So time to revisit.

@scottalanmiller said in Building ELK on CentOS 7:

@dafyre said in Building ELK on CentOS 7:

So... I went through and ran the script and it seems to have worked fine... What next?

Edit: To collect logs from the local server, I also had to install filebeat on this server. So I reckon I can now go and install it on all my other systems as well.

Yes, install Filebeat and point it to ELK. Check my Filebeat article for more info.

Didn't realize you had one. 8-) But I'm good now. Logs are collecting as we speak. Bonus: Fail2Ban and Apache logs also work great in ELK.

@Ambarishrh said:

was wondering the same, they could move all the Linux and save quite a lot of cost

They are "all in" on MS technologies. I followed them when they were building the system. Their sponsor is a 100% MS devotee (he's the father of VBA, for that matter) and there is no way that they would consider something based on logic. They were the pioneer user of the .NET MVC system and everything they have done is based on total lock in to MS, which has its advantages. But overall, they are using costly, slow components to do work. I'm sure that it works pretty well, but as good as it could? No way.

We've seen other communities like that make odd technology decisions leaving them locked in to old schemes and costing a fortune to do what is cheap with modern design choices.

I think, unless you have some crazy log traffic, that if you can get 4GB for ELK in an SMB, you are nearly always good. I'd expect hundreds of servers to be able to log to that, as long as you have fast disks (it still has to get to disk fast enough no matter how much memory there is.)

We've had massive Splunk databases with 32GB - 64GB, but those are taking data from thousands and thousands of servers and doing so as a high availability failover cluster, so they have to ingest, index and replicate in real time.

After some more testing it seems enabling output to journald.conf has worked. I did restart it after I tried that but it didn't show up. Now it's working. Not sure what changed, but at least it's working.

They also forget about SELinux with their CentOS 7 docs. You need sudo setsebool -P httpd_can_network_connect 1 and possibly sudo chcon -R --type=httpd_syscontent_rw_t /opt/kibana

Up and running now.

@JaredBusch said:

I have never successful gotten an ELK server up and running and ingesting logs. I really need to get on this.

Digital Ocean has some great documentation on it. I love having an ELK server without any licensing limitations.

The one really sad part, though, is that it is a single user login out of the box and the user management component Shield is non-free.

Here is the SAR report for the server. Remember we are running at half the cores, half the memory that is recommended - mostly just as an experiment to see how much is really needed for things to be responsive. And so far, ingesting five servers, it is working just fine. We will be adding more servers and keeping an eye on things to see how the performance is and will grow the server if we need to. We are trying to learn from this so that we will have better capacity information. But for a smaller company it looks like a very small server will work just fine. No question that the server is busy, but now that it is up and running and no longer handling the initial setup, it's nowhere near being fully loaded.

02:25:01 PM CPU %user %nice %system %iowait %steal %idle 02:35:01 PM all 12.91 19.61 4.53 0.37 0.00 62.59 02:45:01 PM all 2.68 6.86 2.34 0.20 0.00 87.91 02:55:01 PM all 2.73 6.42 2.25 0.21 0.00 88.40 03:05:01 PM all 2.26 9.77 2.07 0.19 0.00 85.71 03:15:01 PM all 3.56 6.49 2.57 0.30 0.00 87.07 03:25:01 PM all 3.52 12.39 2.90 0.26 0.00 80.93 03:35:01 PM all 2.97 6.45 2.37 0.27 0.00 87.95 03:45:01 PM all 2.54 11.15 2.17 0.17 0.00 83.97 03:55:01 PM all 1.44 5.42 1.69 0.10 0.00 91.35 04:05:02 PM all 0.98 4.86 1.52 0.06 0.00 92.58 04:15:01 PM all 1.54 5.07 1.75 0.09 0.00 91.54 04:25:01 PM all 1.52 10.37 1.91 0.11 0.00 86.10 04:35:01 PM all 3.74 6.99 2.65 0.23 0.00 86.38 04:45:01 PM all 3.11 10.70 2.42 0.24 0.00 83.53 04:55:01 PM all 1.02 5.07 1.59 0.05 0.00 92.26 05:05:01 PM all 1.76 5.64 1.89 0.15 0.00 90.57 05:15:01 PM all 0.93 9.27 1.64 0.05 0.00 88.11 05:25:01 PM all 1.71 5.45 1.86 0.13 0.00 90.85 05:35:01 PM all 2.58 5.40 2.24 0.14 0.00 89.64 05:45:01 PM all 4.18 11.75 2.92 0.25 0.00 80.90 05:55:02 PM all 3.16 5.85 2.13 0.26 0.00 88.60 06:05:01 PM all 3.54 6.36 2.32 0.20 0.00 87.58 06:15:01 PM all 3.14 10.63 2.14 0.16 0.00 83.92 06:25:01 PM all 4.87 11.22 3.27 0.24 0.00 80.40 Average: all 9.22 10.60 3.03 0.41 0.00 76.74

@ajstringham It is very handy! But steep learning curve. I attended a demo conference at the beginning of January up in Orlando. It was very classy and very informational.
They know how to throw a good shin-dig. There is an annual user conference in Las Vegas in October, I think. Analogous to Spice World.