this post was submitted on 25 May 2026
186 points (98.9% liked)

Technology

84918 readers
3444 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 3 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
186
Aggressive AI scrapers are making it kinda suck to run wikis (weirdgloop.org)
submitted 17 hours ago by lemmydividebyzero@reddthat.com to c/technology@lemmy.world
14 comments fedilink hide all child comments
top 14 comments
sorted by: hot top controversial new old
[–] tal@lemmy.today 38 points 13 hours ago* (last edited 13 hours ago) (1 children)

Detect whether the user is a human, but instead of blocking the request (which is going to be obvious to the scraper operator and will just cause the bot developers to go work on better human emulation until they get the data), poison the response. Just as blocking scrapers is hard for website operators, so is separating useful data from not-useful data for people building AI training corpuses.

https://www.cloudflare.com/learning/ai/data-poisoning/

Data poisoning involves injecting malicious information into training datasets to manipulate an AI model's behavior, compromising its accuracy, reliability, and the overall integrity of machine learning results.

[–] ag10n@lemmy.world 7 points 9 hours ago

This is the way. Computer use agents are common and can easily ‘browse’ to a page and grab the content.

[–] tal@lemmy.today 27 points 13 hours ago* (last edited 13 hours ago) (1 children)

The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it's a MediaWiki wiki.

This isn't even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn't create additional load on the wiki servers (better for the wiki operators).

But the bots aren't using them! And on top of that, they aren't even written to follow traditional conventions for being polite in scraping. We've had lots of software spider the Web

that's not new

but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.

It's as if someone has set up a table out front of their house with a big sign reading "Free cookies!" with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.

[–] go_go_gadget@lemmy.world 10 points 11 hours ago

It's a weird phenomenon I encountered as a data engineer inside corporations. Teams wanted to run endless queries off our API instead of using the snapshots we provided. Most of the apps didn't have a justifiable reason for wanting up to the minute information. 🤷‍♂️

[–] CosmoNova@lemmy.world 63 points 16 hours ago (1 children)

Ai scrapers are making it kinda suck to run anything on the internet to be honest.

[–] kalpol@lemmy.ca 5 points 6 hours ago

Making it kinda suck to use the Internet at all. I try to look things up and its page after page of AI slop. We are going to end up back in the days of Altavista.

[–] tate 19 points 16 hours ago (2 children)

I wonder if it would be possible to filter traffic by limiting the rate at which links can be followed after the initial connection.

I suppose one problem with that is it would also limit the crawlers you want, like search engine indexing. Maybe if enough sites were doing this it would generate some pressure on the AI companies to behave better.

[–] Jason2357@lemmy.ca 3 points 13 hours ago (2 children)

The issue with wiki's and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.

[–] bountygiver@lemmy.ml 1 points 1 hour ago

Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?

[–] tal@lemmy.today 2 points 13 hours ago (1 children)

I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.

[–] Jason2357@lemmy.ca 3 points 3 hours ago

Sure. Its just the thousands of obscure page edit history pages that ai crawlers hit every hour that cause the problem.

[–] lemmydividebyzero@reddthat.com 4 points 14 hours ago (1 children)

If they have 10k URLs and in parallel find out all links via different IPs, they get new links.

When they do that again with the new links, there is no connection between visits, because the IPs differ. It looks like someone else is requesting what is behind the link.

[–] bountygiver@lemmy.ml 1 points 1 hour ago

If they have 10K IPs, then you need a DDOS filter, which would require a huge infrastructure no matter what.

[–] carrylex@lemmy.world 2 points 11 hours ago* (last edited 11 hours ago)

Wait a second...I already read this... 2 months ago