The Smol Web

I need to join more communities, because I'm noticing these anti-scraper questions way too late.

I'd like to direct your attention to iocaine. It's somewhat similar to Anubis in the sense that it sits between your reverse proxy and the real content, but unlike Anubis, it does not use proof of work. It exploits the fact that most of the scrapers are incredibly dumb, and can be trivially detected:

• Is it in ai.robots.txt's list? It's a crawler.
• Does it have Firefox/ or Chrome/ in the user agent, but sent no sec-fetch-mode header? Pretty much guaranteed to be a crawler, with few exceptions (eg, Googlebot, Bingbot - but I'd classify those as hostile crawlers too)

Serve garbage or a static page with poisoned URLs to these, and you got rid of 90%+ of the bots. Why the poisoned URLs? Because when they come back riding headless chromes, they usually crawl URLs the dumb bots collected. If you poison those URLs in a way that you can identify them trivially, you can block the headless chromes too, which you wouldn't be able to detect otherwise. Whether they come through residential proxies or not, as long as their queue is collected by the dumb bots, you can catch them.

On top of this, to reduce the load on your servers, iocaine can also block requests. It can be configured to serve garbage & poisoned URLs to the dumb bots, and then firewall anything that hits a poisoned URL.

The false positive rate is surprisingly low.

Short answer: You can't. Longer answer: Accept it if/when it happens, but don't make yourself an attractive target, and don't put yourself in a position where it's going to cost you money if it does. The hype of LLM scrapers is largely overblown for small personal static pages. LLM training wants fresh, data-heavy content. If they are scraping your smolweb site you're either updating it with and hosting rich content far too frequently, or it's an error on their part out of pure ignorance and laziness. That doesn't mean it can't happen, but also, what actual harm does it do to have a dozen scrapers hitting your site every second? (this is an exaggeration it's likely not going to be that bad) How big is your smolweb page and images? A few dozen kilobytes? What's your bandwidth limit, and what happens when you hit the cap? If you're worried about hitting the cap too quickly, this can be straightforwardly managed by per-IP rate limiting and throttling if necessary to keep things under a cap and allow fair access to gentler users. But when you're only hosting small files, most connections have plenty of bandwidth to handle scraping until they realize how pointless it is and give up, and it probably won't be necessary.

I run about 20 small websites, all public and searchable, with no protections at all. Most of them are rarely updated and have been static for years, I just checked my traffic logs for the last day: ~14,000 hits. That may sound like a lot, but for a request that takes milliseconds to deliver, a computer sitting around not doing anything for the many seconds in between each of those requests is probably bored. Many different scrapers are obviously buried in that traffic, but they're not the overwhelming horror that people make them out to be, at least in my experience.

Anubis potentially makes sense on social media sites like Lemmy that are hosting large numbers of users and user-generated content. This stuff is like manna from heaven for LLM bots. Same with code repositories like forgejo. They are very attractive targets for scrapers, with lots of frequent updates that require frequent scraping and also lots of very large files for it to download and ingest. This will absolutely hammer your bandwidth if the scrapers find you an attractive target and they are stupid (which they are).

But smolweb? Honestly, I hate to break it to you but nobody cares that much, not even LLMs.