this post was submitted on 25 May 2026
245 points (99.2% liked)
Technology
84918 readers
3408 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
The issue with wiki's and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.
Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?
That seems to be a lot of peoples approach, but if they cared about time or bandwidth they wouldn't be spidering Dow into your commit history multiple times a day. They have more patience and resources than your human readers.
I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.
Sure. Its just the thousands of obscure page edit history pages that ai crawlers hit every hour that cause the problem.