oefrha

A few sites of mine are backed by SQLite databases replicated daily from another source. Right now I’m rsyncing them, but sending an even increasing number of gigabytes a day isn’t great for the long term. Anyone aware of some sort of checkpointing and incremental sync solution for SQLite? Doesn’t need to be real time, and shouldn’t be real time, really. This one isn’t what I’m looking for unfortunately, the “sign up for a free account” part is also pretty crazy.

pkhuong
Verneuil https://github.com/backtrace-labs/verneuil might work for you. Its replication client can use a persistent chunk cache (no fsync, so doesn't survive reboots), and there's a CLI utility to reconstitute a sqlite db file (`verneuilctl restore`). It also tries to do the right thing by writing to a temporary file before renaming it over the old db file.

On the write-side, you can load the verneuil VFS as a runtime extension. There's a similar gotcha with reboots, where the replication state is persisted but not fsync-ed, so you get a full refresh on reboots. The S3 storage is content addressed though, so you "only" pay for useless bandwidth and API calls, not storage.