Pandoc is great though I struggle with latex. Is there an easier way to go from md to pdf with your own template?

There's a popular template [0] which you can adapt to your needs. I didn't know Latex too, so I cobbled together snippets I found from stackexchange sites [1] (this was before I knew about that template, else I'd have probably started with that)

[0] https://github.com/Wandmalfarbe/pandoc-latex-template

[1] https://learnbyexample.github.io/customizing-pandoc/