Question since you appear familiar with math simulation, could you do what you're describing with Matlab with Octave instead?
Yes.
Easier with Julia, though, because the ODE and PDE support is better and you have better / simpler notation. Performance will be better as well, but for simple problems you can't tell.
For circuits a lot of them are represented by differential-algebraic equations (DAEs) and require modeling tools in order to handle the high differential index of the systems. This is the reason why they are typically handled via acausal modeling systems which can do index reduction. For Julia, this is the ModelingToolkit portion of the SciML ecosystem (https://docs.sciml.ai/ModelingToolkit/stable/), and some modeling tools like https://github.com/ModiaSim/Modia.jl and OpenModelica front-ends https://github.com/OpenModelica/OMJulia.jl.