lemmy.blahaj.zone Also Compromised (talk.kururin.tech)

submitted 1 year ago by Kururin@talk.kururin.tech to c/main@lemmy.ca

18 comments fedilink hide all child comments

The site is down for now but do not try to login into it.

top 14 comments

sorted by: hot top controversial new old

[-] Vampiric_Luma@lemmy.ca 4 points 1 year ago

My EEEEEEEEEEYES AUUUUUUUUUUUUUUUUUUGH the instance is pure white, probably utilizing some witch-craft to bring sunlight directly into my eyes. How do I return the blessed darkness to my screen??

[-] dowath@kbin.social 4 points 1 year ago

Sounds like beehaw.org has shutdown temporarily just to be safe, sounding like a vulnerability in Lemmy.

[-] TruckBC@lemmy.ca 17 points 1 year ago

Currently it seems to be a vulnerability with custom emojis only, which this instance never had, so currently we shouldn't be affected. However this is a developing situation and we will continue to monitor.

[-] zephyreks@lemmy.ca 13 points 1 year ago

I just want to say I love you guys

[-] TruckBC@lemmy.ca 3 points 1 year ago

Thanks!

[-] Arghblarg@lemmy.ca 4 points 1 year ago* (last edited 1 year ago)

Good job admins for staying on top of the situation!

(... and have you made a snapshot backup, just in case? 😁 )

[-] TruckBC@lemmy.ca 7 points 1 year ago

I can't remember exactly but @Shadow@lemmy.ca is the one that takes care of those things, I think he's got it set up to backup every 8 hours? I'm taking a rapid crash course in sysadmin but I'm not really ready to start poking at production server stuff yet.

[-] grte@lemmy.ca 3 points 1 year ago* (last edited 1 year ago)

Seems they're back up and the devs pushed out a patch for the vulnerability.

[-] TruckBC@lemmy.ca 2 points 1 year ago

We have updated to a patched version.

[-] grte@lemmy.ca 5 points 1 year ago

It's been interesting watching this all play out on an open source social network. It's all out in the open so it caused quite the drama, but the actual order of events? Site goes down and is back up and vulnerability patched like 4 hours later? That's really impressive.

[-] TruckBC@lemmy.ca 6 points 1 year ago

Power of the open source community.

In my opinion the "drama" was a critical part of immediately drawing attention to the voulnerabilty and bringing it to the attention of most instance admins very quickly.

Few things that have been added on my to-do list that I've learned from this.

We need more backend man power for coverage.
Major instances, and probably all instances should partner with another instance that's in an opposite time zone for emergency response. Ideally having partnered admins and backend admins with no more than 8 hours difference between each one for 24 hour reliable coverage would be ideal. Partnered admins should in my opinion have each other's phone numbers and have it set to bypass do not disturb.
We need to make sure users know how to contact admins off Lemmy for emergencies, as well as ensure that admins are tagged when a situation like this develops. (To my knowledge no lemmy.ca admin was tagged when this started to unfold.)
There's more thoughts but I can't remember them on 5 hours of sleep 😴

Any additional suggestions are welcome!