This is fantastic - first hand I have noticed this trend is making working with scripting languages less friendly though.
For a Node example, with esbuild, I can no longer ever install packages and assume they will work fine when running under the host (macOS) and Docker VM (Linux).
Unless Pydantic is downloading all OS binaries with the package and loading the right one at runtime, this would become a "problem" as well. For those coming strictly from scripting language land, this could screw up their workflow quite a bit. A lot more of these people have never worked with native binaries than you think.
I, like many here, understand this basic nuance and would rather my tools be fast. However, I think this is quickly eroding the value of using a dynamic scripting language. The platform portability advantage is completely removed as stuff like this spreads and it makes working with those scripting languages a pain in the ass to the point I wouldn't even want to use a scripting language anymore. It becomes totally opaque whether or not your code can actually run on another target.
In my case this is OK, but I think I'm in the minority.
I think that you've been lucky, since even for mostly-pure Python packages, you often can't install in one env and just use it in another, because there might be some deeper dependencies.
My wish would be for there to be a way to get "most of the installation" working in a portable way and for OS-specific stuff to somehow be isolated into a final step, but in the world of Docker images that feels like a rare user story.