I cried in pain when I got to securing my app (uberauth, guardian etc). There's too much to grasp and I ended-up just copy pasting code in frustration.
Elixir & phoenix are fun though. I really like how much easier it is to test an elixir app (functional language).
You can use Coherence. It's basically Devise for Elixir:
For example, when I last read through the codebase it was common to find features that simply wouldn't work, because the code referenced hardcoded parts of the sample application.
You'll also going to run into a lot of issues trying to migrate off of Coherence if you ever need to support anything other than form based username / password login.
I think it really is worth the effort to go with Ueberauth [0]. You'll need to do more work upfront, but the maintainability gains will quickly pay off. You can even use :ueberauth_identy [1] to provide username / password based auth without too much trouble.