https://github.com/rfjakob/earlyoom has worked well for me in the past.