On one hand I appreciate the motivation behind this and feel like engineering for today is an underappreciated skill... but on the other, I can't help but wonder if ELK stack would have been costlier for the load described (< 100 messages a second)
In my experience you're at the sweet spot where ES scaling isn't really an issue, and you get to enable a lot of really useful tooling and exploration for a growing team through Kibana
-
Also for anyone doing this today, ClickHouse now has experimental improved JSON handling: https://clickhouse.com/docs/en/guides/developer/working-with...
The main reason I bring up ELK stack is using materialized views in the style mentioned sounds fairly unergonomic if you have a lot of event types, so with the improved JSON handling in CH you'd at least have a nicer query interface
Author here. The limit we hit with ES was the memory required to have anything approaching decent response times for queries. I've always tried to keep ES in memory but that becomes a pain, and honestly the defaults for CH just... work whereas ES ends up needing a hundred dials slightly tweaked to make it work (and keep working).
The new JSON type looks amazing though! It's probably time I update the code, but honestly it's just been silently chugging along in the background with zero maintenance for now.