Hottest take: is the programming language something that even matters? at all?

Everyone is using the same libraries anyway, there is no blub paradox, we are just talking about which language has shinier bindings.

only true if you are primarily gluing together big libraries and not writing much of your own logic and abstractions.

We are talking about ML. Nobody writes their own symbolic differentiation libraries or LAPACK.

They do, just not if they're using python. In Julia they absolutely do. Here's a matmul and LU respectively that are SOTA (especially for small matrices) https://github.com/JuliaLinearAlgebra/Octavian.jl and https://github.com/JuliaLinearAlgebra/RecursiveFactorization...