The_rationalist

Does it leverage deepspeed/zero 3?

minimaxir

That’s PyTorch only; the current models are TensorFlow.

The_rationalist

Oh that's unfortunate, can't the models be exported to pytorch through e.g onnx?

stellaathena
There's a PyTorch + DeepSpeed repository here: https://github.com/EleutherAI/gpt-neox