Only on par with GPT2-XL, was not trained to follow instructions.

Actually, that is changing: https://github.com/BlinkDL/ChatRWKV