Disclaimer: I work for Lightrun, a dynamic instrumentation (read: add logs at runtime) tool.

Logging is a surprisingly pricey bit of observability at scale.

If you're doing anything highly-transactional you're pretty much guaranteed nowadays to get major observability bills (for things like ingestion, transmission, storage and analysis/querying).

It also creeps up on you - you're used to getting billed incessantly for cloud stuff, monitoring doesn't sound too pricey. Until, well, it is.

Can't help but point out that this is static logging, meaning logging added during development. This stems from an the approach colloquially referred to as "log everything, analyze later", rather than more disciplined, case-specific logging.

We're used to add "just-in-case-logs" to ensure that we've got ourselves covered when the sh*t hits the fan, but we rarely look at them.

An alternative approach would be using a tool, like the one Lightrun[0] builds (see disclaimer above) to add Logs in real-time to applications.

This means that instead of logging IN ADVANCE (i.e. during development) you can add logs when and where you need them, in real-time. The tool works using a variety of dynamic instrumentation techniques (depends on the runtime), and currently supports Java, Node.js & Python (.NET soon).

It can also pipe these logs into Loki, Elastic or SigNoz since they can be dumped like normal logs, right into the stdout.

In any case, we're seeing mass reduction in static logging (up to 60% in volume, 40% in cost) by going dynamic instead of mostly static. You'll always have to log some stuff for forensics and traceability, but dynamic instrumentation removes a lot of the dead weight.

[0] https://lightrun.com