EDIT: Judging by the upvotes, my issue below might be a common problem. Read below and be warned :)
---
Well I guess they did say Beta...
I turned this on immediately and now get HTTP 500 errors on every page on GitHub while logged in. Clear cache and login again does not solve.
Oh dear.
Thankfully their support portal does still work. For anyone as unlucky as me:
I just get 500 on the homepage, and my profile also 500s when logged out. Most other things still seem to work, including https://github.com/settings/profile where I could turn it off.