In 2024 I co-built a Telegram platform whose whole business depended on one thing: reliably pulling engagement data off Twitter, thousands of records a day, for campaigns that paid people based on those numbers. If the scraper lied, people got paid wrong. If it died, nobody got paid.
The first version ran flawlessly on my laptop for a week. I shipped it to a server on a Friday. By Saturday afternoon the success rate was somewhere around a third, and by Sunday the account pool was mostly locked. I spent the next few months learning, the hard way, what I'm about to compress into a few hundred words. Then I went and got a job at Scrapfly, where this is the entire job, and discovered how much I'd still gotten wrong. The irony is not lost on me.
You're caught before you send a single byte of HTML
The mistake everyone makes first, including me: obsessing over the User-Agent header. I had
a beautiful rotating list of them. It didn't matter, because I was being fingerprinted during the
TLS handshake, before the HTTP request even existed.
Every client announces itself when it opens a TLS connection: which cipher suites it supports,
in what order, which extensions, which curves. Hash that and you get a fingerprint (JA3, and now
its successor JA4). Python's requests has one. httpx has one. Chrome has a very different one.
So my request said "I am Chrome 124 on Windows" in a header, while the handshake underneath it
screamed "I am a Python script." That contradiction alone is enough to get scored as a bot.
The fix is to make the handshake match the story:
from curl_cffi import requests
# Impersonate a real browser's TLS stack, not just its headers
r = requests.get(url, impersonate="chrome")
Or, if you need JavaScript anyway, use a real browser and let it do its own honest handshake. Which brings up the second lesson.
Consistency beats cleverness
My next version rotated everything on every request: proxy, user agent, viewport, timezone, WebGL vendor. I thought I was being thorough. What the anti-bot system saw was one session whose device changed six times in ten minutes. Nothing real does that. I'd built a stack of spoofing that was fighting itself.
The thing that actually worked was boring: pick one coherent identity per session and keep it. Timezone that matches the proxy's geography. Language headers that match the timezone. Screen size that exists. Then don't touch it until the session ends. Minimal changes, maximum coherence. Look like a slightly dull person, not a very clever robot.
Sessions are the difference between scraping and browsing
The single biggest jump in success rate came from cookie persistence. A fresh browser context on every request is a stranger showing up at the door every time. A returning visitor with a cookie jar, local storage, and a history of behaving normally gets a much easier ride.
In Playwright this is almost embarrassingly little code:
# End of a good session: save who we were
await context.storage_state(path=f"sessions/{account_id}.json")
# Next run: be that person again
context = await browser.new_context(storage_state=f"sessions/{account_id}.json")
Pair that with a stable fingerprint per account and you've gone from "suspicious new device every five minutes" to "the same laptop that logged in last Tuesday." That's most of the 95% success rate we eventually held.
Back off when the site escalates
The last hard-won lesson: a retry loop is a confession. When a site starts serving challenges, hammering it with retries from the same identity tells it exactly what you are. I replaced retries with a small circuit breaker per account: track the rolling success rate, and when it drops below a threshold, stop. Park the account for hours. Let a different one work. Come back looking like someone who took a break, because that's what people do.
Two years later
The arms race has moved further down the stack. Post-quantum key exchange is now default on big CDNs, and whether your client even offers it is a detection signal. JA4 is in production at basically every anti-bot vendor. The specific tricks I used in 2024 have a shelf life measured in months.
But the mental model has held up completely: be consistent, be boring, be a returning visitor, and know when to leave. Everything else is implementation detail that will be outdated by the time you read this.
If you're building a scraper today and it works perfectly on your laptop, I have news for you. Deploy it on a Friday. You'll learn everything above by Sunday.