Rather than start a new thread, I'll just mention that Twitter revised its API policy changes (the relevance to this thread should be obvious): https://twitter.com/TwitterDev/status/1623467615539859456
Key points:
- Free API continues until Feb 13 instead of sunsetting tomorrow
- $100 paid tier grants access to Ads API, which is where promotion and scheduling of Tweets happen. So this will become a necessary but minor expense for brands, influencers, and third parties that cater to them.
- Free API access will continue to be available but users will be limited to 1500 tweets/month.
This actually strikes me as a rational compromise that will bring in a few million $ a month in revenue with significantly disrupting the development/research/comedy bot ecosystem. Bad for spammers and shit-tier brands, both of whom deserve to suffer anyway.
100$/Mo is a lot for most and Twitter is relatively easy to scrape and script.
I'd imagine this move will probably not earn Twitter any considerable amount of profits as Twitter is weirdly heavy app and new lazy web scrapers/bots will drain all of those 100$ sales.
For some context, currently there are two ways to scrape/automate twitter outside of the official API since it's a heavy JS-only web app:
- run a real script-controlled browser (using something like Playwright/Selenium/Puppeteer).
- reverse engineer their backend graphql API.
The former is super easy to do but since it runs a full web browser it's expensive resource-wise as it loads everything a normal web user would load (unless configured explicitly not to). The latter is much more efficient but not accessible for most developers as Twitter's backend is pretty complex.
Isn’t GraphQL self describing?
To add, Twitter's backend has a bunch of layers on top of it. I did a bit of reverse engineering for my blog article on scraping Twitter [1] and there's a bit of magic token/header generation involved.
Good resource if you'd like to learn more about how the whole backend works is Nitter [2] which implements most graphql functions in Nim (it's very readable even if you don't know Nim).
1 - https://scrapfly.io/blog/how-to-scrape-twitter/#scraping-pro...