My particular needs for PDF creation are somewhat unusual, so I'm not surprised that Acrobat Pro seems to be the only thing that suits my needs. If anyone knows of a PDF creation library that has good support for the various image compression schemes supported by PDF, I'd love to know about it.

Currently I'm using a 10 year old paid version of Acrobat Pro because I'm unwilling to spend another $500 to get a modern copy simply to compress images. I use it for scanning and archiving old computer manuals. In many cases jbig on 1bpp images is good enough; other times I want to use one of the more sophisticated hybrid schemes so I can have B&W or color images interspersed on the page with 1bpp text. Acrobat does a pretty good job of auto-deskewing images and doing analysis to figure out which compression type to use in different sections of the same page.

https://github.com/4lex4/scantailor-advanced

I find the autodeskewing algorithm to work well, but it allows hand adjustment as well, which I like. As I've gotten better as using it, I've been able to get the size of my scanned documents down considerably by cleaning up the scans. This includes some old manuals.

As far as the pdf encoding itself, I use both mutool, from mupdf, and qpdf. I just checked and it looks like while they both compress their streams, it may not have the same flexibility with Acrobat Pro. For me, I'll decompress, edit, and recompress streams on the files and that's been fine for my use.

That said, if someone knows of a better tool for compressing streams in a PDF, I'd be interested to hear about it as well.