Are there any container filesystems that support multiple inheritance, and create diff layers? It would be really nice if I could build a few different things independently, and then merge the final images together. Also if I could only include files that have changed in a new layer, and ignore duplicate files (even if the file was touched, or the timestamp has changed.)

Those are my biggest pain points with Docker at the moment. I have a complex build script that uses multi-stage builds and rsync to achieve this [1], but it's still a bit slow and inefficient. Would be nice if something supported this out of the box.

I've worked on a lot of projects where people just reinstall (and recompile) their entire list of dependencies (Ruby gems or NPM packages), and you have to jump through hoops to set up a caching layer, or maybe install them into a volume as a run-time step, instead of at build time. There should be a much better native solution for this, instead of needing to invent your own thing or read random blog posts.

[1] https://formapi.io/blog/posts/fast-docker-builds-for-rails-a...

I would say check out buildkit, which is the tech behind "docker build"'s new builder.

I don't know if the Dockerfile format is really suitable for this, but you can now build your own format and Docker can just build it.

Basically buldkit breaks things down into a frontend format (like Dockerfile) and a frontend parser which gets specified as an image at the top of your file (`#syntax=`), the parser converts the frontend format into an intermediary language (called llb), buildkit takes the llb and passes it to a backend worker.

This all happens behind the scenes with `DOCKER_BUILDKIT=1 docker build -t myImage .`

Docker actually ships new Dockerfile features that aren't tied to a docker version this way.

Actually there are a number of new Dockerfile features that might get you what you need, even if the format isn't all that great, at least it's relatively natural to reason about. Things like cache mounts, secrets, mounting (not copying) images into a build stage's "RUN" directive, lots of great stuff.

This is all officially supported stuff.

Here's a demo of "docker build" building from a buildpack spec instead of Dockerfile: https://github.com/tonistiigi/buildkit-pack

- buildkit - https://github.com/moby/buildkit - official Docker docs - https://docs.docker.com/develop/develop-images/build_enhance... - buildkit Dockerfile docs - https://github.com/moby/buildkit/blob/master/frontend/docker...