What does HackerNews think of verneuil?

Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.

Language: C

Very cool, I'd have a big use case for "Litestream in Rust" as a library.

Any way to get notified about the release?

Also: will this be a straight port, or could it also support "live-replication" from S3, instead of just on-demand restores? https://github.com/backtrace-labs/verneuil supports this. Sadly it isn't very actively maintained and buggy.

Verneuil https://github.com/backtrace-labs/verneuil might work for you. Its replication client can use a persistent chunk cache (no fsync, so doesn't survive reboots), and there's a CLI utility to reconstitute a sqlite db file (`verneuilctl restore`). It also tries to do the right thing by writing to a temporary file before renaming it over the old db file.

On the write-side, you can load the verneuil VFS as a runtime extension. There's a similar gotcha with reboots, where the replication state is persisted but not fsync-ed, so you get a full refresh on reboots. The S3 storage is content addressed though, so you "only" pay for useless bandwidth and API calls, not storage.

https://github.com/backtrace-labs/verneuil doesn't even need an HTTP connection between the writer and readers: readers only interact with the object store.
Verneuil (https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem.

I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba...

(the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...)

Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12

https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand.

Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!