Mostly good stuff but a few comments:

- article doesn’t clarify if it’s on hardware or VMs

- 140 shards per node is certainly on the low side, one can easily scale to 500+ per node (if most shards are small, typically power law distribution)

- more RAM is better, and there is a ratio of disk:ram that you need to keep in mind (30-40 for hot data, 200-300 for warm data)

- heaps beyond 32g can be beneficial but you’d have to go for 64g+, 32-48g is a dead zone

- not a single line about GC tuning (I find default CMS to be quite horrible even in recommended ~31g sizes)

- CPUs are often a bottleneck when using SSD drives

Serious question: does indexing Logstash/JSON logs really need to take gigabytes of memory + disk and sharding?

No. ELK is a slow and expensive way to store and retrieve logs. The reason people use it is that nothing else exists. (I was blown away when I started using it at my last job. I used the fluentd Kubernetes daemonset to extract logs from k8s and into ES... and the "cat a file and send it over the network" thing uses 300+MB of RAM per node. There is an alternate daemon that can be used now, but wow. 300MB to tail some files and parse JSON.)

I think a better strategy is to store logs in flat files with several replicas. Do your metric generation in realtime, regexing a bunch of logs on a bunch of workers as they come in. (I handled > 250MB/s on less than one Google production machine, though did eventually shard it up for better schedulability and disaster resilience. Also those 10Gb NICs start to feel slow when a bunch of log sources come back after a power outage!)

For simple lookups like "show me all the logs in the last 5 minutes", you can maintain an index of timestamp -> log file in a standard database, and do additional filtering on whatever is retrieving the files. You can also probably afford to index other things, like x-request-id and maybe a trigram index of messages, and actually be able to debug full request cycles in a handful of milliseconds when necessary. For complicated queries, you can just mapreduce it. After using ES, you will also be impressed at how fast grepping a flat file for your search term is.

The problem is, the machinery to do this easily doesn't exist. Everything is designed for average performance at large scale, instead of great performance at medium scale. Someday I plan to fix this, but I just don't see a business case, so it's low priority. Would you fund someone who can make your log queries faster? Nope. "Next time we won't have a production issue that the logs will help debug." And so, there's nothing good.

Maybe https://github.com/grafana/loki , but haven't yet tried it.

(Or https://github.com/phaistos-networks/TANK ..?)

> I think a better strategy is to store logs in flat files with several replicas

Agreed. We just used beats + logstash and put the files into Ceph.

> x-request-id and maybe a trigram index of messages, and actually be able to debug full request cycles in a handful of milliseconds when necessary.

Yes, yes, yes. That would be great.