Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.
Do you use ts_rank? PostgreSQL FTS is very efficient until you want to rank the results according to their relevance. This is because the data necessary to the ranking are not in the GIN or GIST index. They are in the heap, and this triggers a lot of random IOs.
Ah, this is good to know. My site doesn't yet need to scale, so this is definitely A Problem I Would Love To Have ;)
EDIT: This seems to help with the ranking problem: https://github.com/postgrespro/rum