Earlier in the series:
- We are losing our host and will need to move to a VPS.
- We moved to object storage for images over last weekend.
- On Friday we moved to the new host
TL;DR of this post: We now have Anubis on Lemmy.nz, which helps protect against scrapers.
Hi everyone! As some have noticed, the site has been slow these past few days, which was due to a massive amount of traffic presumed to be from AI scraping bots training LLMs. I posted this elsewhere but will put it here again for good measure (7 day history):
Yeah so that peak of the graph is over 2 million unique requests that day, and the earlier part of the week has about 100k a day. So a 20x increase in traffic.
You may have also seen some weird stuff over this weekend as I played with how to get Anubis inserted into our Lemmy setup, which wasn't too straightforward with our multitude of frontends and our load balanced Lemmy-UI. You might have seen errors, apps might not have been able to load due to being incorrectly blocked, or you might not have been able to load the site at all. Apologies for this, but I think I have it sorted now.
Running under the assumption that the scrapers are not directly targeting Lemmy but are just scraping everything on the net and we just happen to have a significant amount of content, Anubis has been added only to the front ends (websites you see accessing in a browser), and if you are logged in then the policies are set so Anubis should just pass you straight through without any checks.
The APIs and federation and anything else happening directly with the backend do not go through Anubis. This means apps should not be affected and loading images should not be affected.
I've already noticed a big difference in CPU usage. It went from periods of time averaging over 150% (as in, the CPU couldn't keep up, and you'd see Lemmy fail to load at these times), to the last hour peaking at 47% and mostly hovering around 15-25% CPU usage.
Click for image of last 24 hours CPU usage
Adding Anubis is a pretty big change that has a decent chance of breaking something so please let me know if you see anything weird!
I'm also happy to give more technical info if anyone wants to know about how it's implemented, the architecture of our Lemmy setup and how it fits in, or anything else. Just ask!
