this post was submitted on 25 May 2026
213 points (99.1% liked)
Technology
84918 readers
3382 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it's a MediaWiki wiki.
This isn't even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn't create additional load on the wiki servers (better for the wiki operators).
But the bots aren't using them! And on top of that, they aren't even written to follow traditional conventions for being polite in scraping. We've had lots of software spider the Web
that's not new
but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.
It's as if someone has set up a table out front of their house with a big sign reading "Free cookies!" with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.
That compressed database would have to be updated frequently, I don't know how well it'd work.
Wikipedia has started striking deals with AI companies as a means to cover the cost of their lookups which, as much as I may not like it, is totally fair.
A wiki for some obscure indie game won't have the same leverage.
They are.
It's a weird phenomenon I encountered as a data engineer inside corporations. Teams wanted to run endless queries off our API instead of using the snapshots we provided. Most of the apps didn't have a justifiable reason for wanting up to the minute information. 🤷♂️