I have been doing this for years, but for me the biggest downside is the lack of API at my financial institutions to pull data , which leads to a big time suck weekly/monthly logging into sites , navigating their poor UI to download CSVs.
Otherwise there is a great ecosystem of tools (and I have automated a lot of my own with Python) for parsing, classification, and import of data into the plain text ledgers.
ledger-autosync [1] tries to solve this problem, by pulling the data for you from supporting banks. If I understand things correctly, this [2] is the list of banks it can successfully pull data from.
[1] https://github.com/egh/ledger-autosync [2] http://www.ofxhome.com/index.php/home/directory/successfulRe...