I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, memory optimal storage for this.
RocksDB seems to fit few boxes but there could be much better solution as we don't need deletes/range scans sort of operations.
Any suggestions?
Check how https://github.com/netdata/netdata does it.