Elastic Search

What we like most about Elasticsearch is its incredible speed and scalability. Its distributed architecture allows us to handle petabytes of data and perform complex queries in milliseconds, which is absolutely critical for the real-time needs of our marketing orchestration product. The powerful aggregation framework is another favorite, as it enables our engineers to build highly analytical features and dashboards with relative ease.

It can aggregate logs, and it has a lot of flexibility in sorting and visibility.

I like its ability to handle large-scale data with high performance and near real-time search capabilities. The flexibility of the query DSL, powerful aggregations and developer friendliness with an amazing support team from Elastic made it a success story in every project I used it. With it supoorting vectors storage is an additional feather it its hat allowing us to build RAGs on top of it.

Version Upgrades: We've found that upgrading to new major versions can be a painful process. It often involves schema changes or breaking API changes that require significant refactoring and testing, consuming valuable development time. Resource Intensity: Elasticsearch is resource-hungry. To achieve the low latency and high throughput our product demands, we have to allocate a large amount of memory and CPU, which drives up our cloud infrastructure costs. Complex Performance Tuning: It can be difficult to diagnose the root cause of performance issues. We've spent considerable time fine-tuning settings and re-indexing data to address specific query performance problems, which highlights the complexity behind its seemingly simple interface.

We use the endpoint for AV on servers, and it can be a bit of a pain managing what it is doing. The clients we use for log aggregation alone on user devices are pretty heavy. We are literally seeing battery life cut by 1/3 to 1/2 on mobile devices with the client installed. I can use the product for log searching, but it is very manual (I am a sys admin). I honestly usually just pull logs locally. The security team picked the product, and they are not much better at using it than me. I have seen one ticket generated due to the output of elastic, and it was because the server AV had latched onto the build process we were using to compile our product. It took almost a week before they either got around to fixing it or figured out how to fix it (they are pretty closed-door, so I am never sure what is going on.) Also it is VERY expensive, especially if you use their cloud services. On top of that, the security team is looking at more products to tie into Elastic to make it easier to use. This will of course cost more money.

Back in 2015, it was mostly about managing operational complexity at scale. Finding those specific config options to tune the cluster and optimize the resources. But as it got more widely adopted and documentation increased, along with a great support team from Elastic, the problems went away.