FYI: lemmy.ml - central hub for many communities - has been down 2 hours after 0.18 upgrade EDIT: back up - they say DDoS attack against multiple Lemmy instances (sh.itjust.works)

submitted 1 year ago* (last edited 1 year ago) by RoundSparrow@sh.itjust.works to c/main@sh.itjust.works

10 comments fedilink hide all child comments

Based on my weeks of code (and operating instance) study on lemmy_server, I strongly suspect that peer servers upgrading to 0.18 is causing a swarm of federation activity that lemmy.ml is getting as peers go down and back up.

you are viewing a single comment's thread
view the rest of the comments

[-] SJ_Zero@lemmy.fbxl.net 7 points 1 year ago

oooooh, probably all those messages that never went through are starting to slog through the system, and a million people worth of backlog are getting slogged through.

[-] RoundSparrow@sh.itjust.works 12 points 1 year ago

Lemmy 0.18 (and earlier) has no ability to save the federation queue, when the server stops it forgets what it had in retry to send. I suspect the problem is more the other servers going down causing lemmy.ml to keep crashing on outbound content it is sending. There are hundreds of small servers out there subscribed to lemmy.ml's communities.

[-] agressivelyPassive@feddit.de 8 points 1 year ago

Seems like a rather bad design, if a long queue is neither saved nor sufficiently isolated to not disturb the actual usage.

[-] SJ_Zero@lemmy.fbxl.net 5 points 1 year ago

I think I was already told that, so I should know better.

Still pretty interesting. I'd give you a thumbs up emote, but I haven't upgraded yet. :P

this post was submitted on 23 Jun 2023

31 points (100.0% liked)

sh.itjust.works Main Community

7584 readers

1 users here now

Home of the sh.itjust.works instance.

founded 1 year ago

MODERATORS

TheDude@sh.itjust.works

manifex@sh.itjust.works

biela@sh.itjust.works

imaqtpie@sh.itjust.works