Question : I'm building a 3d procedural geometry generator and I need to store and index it by tiles for hundreds of kilometers.
I'm curious if it can be efficient to store polygonal data in a database instead of a dedicated file format.
I'm never sure how to properly copy raw struct binary data from sql, directly in ram. I think avoiding parsing the text output result in better performance, but I'm not entirely sure...
Options include the https://sqlite.org/rtree.html module, and building an application-specific mapping from geometric indexes to an integer keyspace (https://github.com/google/s2geometry or similar).
We're using SQLite archives of many GB successfully without issue. As long as the primary keyspace is well-designed (see also https://sqlite.org/withoutrowid.html), ranged queries are extremely fast.
> I'm never sure how to properly copy raw struct binary data from sql, directly in ram.
BLOB columns and an application-specific serialization/deserialization step work well. memcpy to a struct works if you are absolutely certain that you know what the layout will be. All of the standard perils apply - alignment, internal padding, platform-specific layout, endianness, etc.
We're using Protobuf with success. I imagine Flatbuffers would also work well. I'd put Protobuf/Flatbuf and their competitors on the front of the tool shelf.