I wonder if there are any search engines dedicated to indexing these kinds of libraries. I know there's a decent one just for scihub, but it would be awesome if I could do a Google-style search that returned the contents of books, magazines and journal articles instead of just websites.

Wasn't that what google books was supposed to be?

Does anyone know of a good FOSS alternative for google books that could be self-hosted (for personal library)?

Does Calibre fit the bill? The program itself is great, but it supports a plugin system that really puts it over the top. One of them automagically strips off Adobe DRM for any book loaded in to it.

https://calibre-ebook.com/

Seems that with calibre-web[1] we can have nice web front-end. But it doesn't have full-text search inside the content of the books though, unlike google books.

[1] https://github.com/janeczku/calibre-web