LemmyToday

321 readers

2 users here now

If you experience issues or problems with this instance (lemmy.today), this is the place to discuss them. Or if you just want to ask questions about how something works. Anything related to the instance or lemmy itself.

founded 2 years ago

MODERATORS

mrmanager@lemmy.today

Testing server load with login required on old.lemmy.today (lemmy.today)

submitted 1 day ago* (last edited 1 day ago) by mrmanager@lemmy.today to c/lemmytoday@lemmy.today

16 comments fedilink hide all child comments

Hi guys,

Im doing some saturday testing for old.lemmy.today again.

Currently login is required for old.lemmy.today to read posts there. If you are already logged in, you wont notice any difference.

The reason im experimenting is because bots are hammering it, and I wanted to get some statistics of load if I simply dont allow anonymous scraping. Some bots are passing Anubis and scraping content.

So just log in if you havent and it will work just as before.

Thank you!

you are viewing a single comment's thread
view the rest of the comments

[–] mrmanager@lemmy.today 3 points 1 day ago

Also, while improving server performance and failure modes under load would help, the “badly-written, aggressive scraper bots that ignore robots.txt and are given loads of network resources clobbering servers” is something that affects many, many different Web servers out there. This isn’t a Lemmy problem, nor even just social media problem. It’s a Web problem.

100%. Everyone gets this problem and then they put their site behind cloudflare. But what happens when the entire Internet is behind an american company who can decide which sites it likes and which it doesnt?

Same thing for the Threadiverse. The Threadiverse will let you set up a Threadiverse instance and subscribe to everything, efficiently feed all the posts and comments you want to your instance, the moment they come in. In nice, machine-readable form, rather than in something intended for humans that you have to scrape and post-process. But…it takes more dev effort to set up something specific to the Threadiverse than to just treat it like another website.

Yeah, the web was intended to be this universal format that would always work on any platform and any operating system. But the downside is clear today. Some estimates say that we already have much more bot activity than human activity on the internet. And we have AI creating content for the last few years...