Why not downloading the html page and scrape that file instead of using puppeteer (which uses a headless chrome I think) ?

I didn't have an easy solution to grab that JSON variable directly by scraping HTML. So I chose Puppeteer because it simulates the browser (yes, chrome) and therefore I could just get the JSON object with a bunch of lines as if I was using the browser's console. It's slow, but I think it's the most straightforward way to do it. But if you could elaborate more on your approach I will definitely try that out!

grose

I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM.

Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site.

https://github.com/PuerkitoBio/goquery https://github.com/robertkrimen/otto