Elastic Observability

I've been using Elasticsearch since early days, with very different things in mind.

I started with simple text search - with some additional tweaks, stemming and other cool features it helped us drive enormous traffic to our website. I can't imaging pulling it off so easily with any other tool.

Every day I use it for web server log analytics. Search and great visualizations make it an absolute essential in work my toolset.

We also run a lot of algorithm analytics on top of our Elasticsearch cluster.

If you're looking for managed options check AWS Elasticsearch Service, or the recently introduced Elastic Cloud.

My only concern with Elasticsearch is that it might get expensive to run pretty quickly. But with a certain amount of effort put into optimization it's gonna be worth it.

The boards are handy and show all the required information at once

The query language is proprietary and difficult to master

Elastic Search is easy to install
Easy to use as there are loads of documentations present online
Easy to scale up when the need arises
Uses REST FULL API which is light weight.

monitoring of Elastic Search are complex like wise administration
Installation of elastic search on windows OS is not straight forward

Logstash does a great job of ingesting data, transforming it and then storing in a 'stash' like elasticsearch. It can virtually ingest any data as long as you have the time and patience to set it all up. We use it as part of the process of collecting all logs from a multitude of sources using filebeat (generally application and system logs) on different operating systems, shipping to logstash then transforming and storing that in elasticsearch and having Kibana for dashboards and analysis. It works great for this use case.

It can be tricky to set up as it is mostly just a command line tool with configuration files. The recent log4j issues also make you think again about all this log shipping, transformation and ingesting in a different way. If there is similar vulnerabilities in the future, it could end up happening way down in your process in non-public components due to the way logs are shipped and processed down the line.

Elasticsearch has turned into a solution that you'd typically integrate into another to a fully customizable standalone solution. It has everything in one spot now, with hardly a need to use 3rd party plugins/extensions.

The licensing models are very confusing. There is a big push to get on their hosted SaaS offering but that itself is also not very clear.