Elastic Search

What we like most about Elasticsearch is its incredible speed and scalability. Its distributed architecture allows us to handle petabytes of data and perform complex queries in milliseconds, which is absolutely critical for the real-time needs of our marketing orchestration product. The powerful aggregation framework is another favorite, as it enables our engineers to build highly analytical features and dashboards with relative ease.

It can aggregate logs, and it has a lot of flexibility in sorting and visibility.

1) Performance and scale: Sub-second queries across large, constantly changing indexes, even during peak ingestion windows 2) Search quality: Fuzzy matching, relevance scoring, and geospatial queries give us precise control to build Find exactly what I meant experiences 3) Flexibility and ecosystem: The ELK stack plus ingest pipelines, analyzers, and ML features make it easy to onboard new data sources and iterate quickly

Version Upgrades: We've found that upgrading to new major versions can be a painful process. It often involves schema changes or breaking API changes that require significant refactoring and testing, consuming valuable development time. Resource Intensity: Elasticsearch is resource-hungry. To achieve the low latency and high throughput our product demands, we have to allocate a large amount of memory and CPU, which drives up our cloud infrastructure costs. Complex Performance Tuning: It can be difficult to diagnose the root cause of performance issues. We've spent considerable time fine-tuning settings and re-indexing data to address specific query performance problems, which highlights the complexity behind its seemingly simple interface.

We use the endpoint for AV on servers, and it can be a bit of a pain managing what it is doing. The clients we use for log aggregation alone on user devices are pretty heavy. We are literally seeing battery life cut by 1/3 to 1/2 on mobile devices with the client installed. I can use the product for log searching, but it is very manual (I am a sys admin). I honestly usually just pull logs locally. The security team picked the product, and they are not much better at using it than me. I have seen one ticket generated due to the output of elastic, and it was because the server AV had latched onto the build process we were using to compile our product. It took almost a week before they either got around to fixing it or figured out how to fix it (they are pretty closed-door, so I am never sure what is going on.) Also it is VERY expensive, especially if you use their cloud services. On top of that, the security team is looking at more products to tie into Elastic to make it easier to use. This will of course cost more money.

1) Operational Complexity: Capacity planning, shard/replica tuning, and rolling upgrades require experienced hands or a move to a managed cloud 2) Cost governance: It's easy to turn on premium features and ingest more data than needed; without guardrails, costs can creep 3) Dashboarding constraints: Kibana is strong for ops, but highly bespoke business dashboards may require complementary BI tools