What does HackerNews think of scrapy-splash?

Scrapy+Splash for JavaScript integration

Language: Python

Show HN: Flyscrape – A standalone and scriptable web scraper in Go | Nov 2023

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash (https://github.com/scrapy-plugins/scrapy-splash).

Rendering AJAX-crawling pages | Dec 2017

Expand Context ↕

I think the good people at Scrapy [1] have done a great job in keeping pace with making open-source crawler components that handle javascript. It is called Splash [2], and I use it for rendering out screenshots.

[1] - https://scrapy.org

[2] - https://github.com/scrapy-plugins/scrapy-splash

Ask HN: What are best tools for web scraping? | Nov 2017

Expand Context ↕

Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash.

[0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-...

[1] https://splash.readthedocs.io/en/stable/index.html

[2] https://github.com/scrapy-plugins/scrapy-splash

Web Scraping: Bypassing “403 Forbidden,” captchas, and more | Mar 2017

Expand Context ↕

Splash https://github.com/scrapy-plugins/scrapy-splash

Runs a little headless browser.