Recent issues appear to be AI bots scraping lemmy content, posing as legitimate user traffic.
The difference between this and a regular DDoS is the intent behind it. I don't believe the traffic we're seeing is intended to be malicious as with a DDoS, but due to the incompetence of those behind the scraping... it effectively is a DDoS.
Cloudflare's security features are serving us very well:
The red line shows the blocks put in place stopping a recent surge in hits. If the blocks weren't in place, AZ would likely be almost entirely offline.
It works without Internet too...
Thanks, that confirms some of the changes I've made are working.
Thanks for posting, good to confirm it's working as expected.
I've made further changes... what app do you usually use? Worth trying again now, please let me know how it goes.
I'm still working to relax the restrictions.. unfortunately it's very difficult to differentiate legitimate users traffic from the bots.
Since you guys are so active.. how has the site felt for the last hour or so? Any issues?
How is everyone finding AZ performance for the last hour or so?
I've expanded the cloudflare challenges to all non Australian requests. This has made a massive impact.. I'll look at ways to relax this tonight.
For now users outside Australia will receive a cloudflare challenge in their browser. Lemmy apps will likely not function.
Yes, amongst other things.
We've seen some issues with AZ again today. On investigating it looks like more DDOS/abuse traffic from overseas. I've applied the same sort of block as last time we saw this, and things appear to have returned to normal..
The sharp drop is when the "attack" started, the jump is when it was blocked... the red line is blocked requests.
Its been 6 months or so... figure its time for another of these. Keep in mind there have been some major config changes in the last week, which has resulted in the oddities below.
Graphs below cover 2 months, except Cloudflare which only goes to 30 days on free accounts.
CPU:
Memory:
Network:
Storage:
Cloudflare caching:
Comments: The server is still happily chugging along. Looking even happier now that I've properly migrated pict-rs to its integrated object storage config, rather than the bodged up setup.
RAM/CPU are all fine. Storage use is growing slowly as various databases grow. Still a long way from needing to purge old posts, if ever.
Cloudflare is saving less traffic these days, since Lemmy added support to proxy all images. Not a concern, well under the bandwidth cap for the server.
As usual feel free to ask any questions.
In future best to not post referral links, or at least disclose this in your commentary.