63
submitted 1 year ago* (last edited 1 year ago) by ticoombs@reddthat.com to c/reddthat@reddthat.com

Unfortunately we had an issue updating to the latest version of Lemmy. This caused database corruption and we had to roll back to a backup that was unfortunately from 2 days ago!

An abundance of caution in the wrong error resulted in me wanting to push this change out to help with the exploit relating to custom emoji. As other instances were attacked, I didn't want reddthat to fall to the same issue.

I am so sorry !

I have since updated our deployment scripts to automatically take a back up before all updates, as well as investigating why our cron backups didn't work as we expected them too.

We also lost all new accounts that were created over the past 2 days, all new posts, and all subscriptions. I'm very sorry about that and hope that you will reregister with those account details again, repost the memes and pictures that make our server great.

The good news is that we have updated to the latest version of Lemmy and everything is back to "normal".

Fixing Deployments

I'm going to use some of our money for a dedicated development server as our current deployment testing is clearly lacking.
Unfortunately that will have to wait until the weekend when I have some extra time to put into developing these new processes.

Tiff

top 14 comments
sorted by: hot top controversial new old
[-] MentallyExhausted@reddthat.com 22 points 1 year ago

That sucks, but thanks for the transparency, Tiff. Good call on the auto-backup and dev environment, cool that you have the resources to do that now.

[-] ticoombs@reddthat.com 10 points 1 year ago

I've sorted the auto-backup now!

[-] lulztard@reddthat.com 18 points 1 year ago

Growing pains. Part of me adores how janky the Fediverse still is. :D

[-] djsaskdja@reddthat.com 16 points 1 year ago

It’s all good brother cheers from the fediverse

[-] Otakat@reddthat.com 12 points 1 year ago

Two days is nothing compared to the misery that would come from either a compromised server or a botched upgrade.

[-] ticoombs@reddthat.com 6 points 1 year ago

Yes. Unfortunately it should never have happened. But at least it was only 2 days, and not weeks!

[-] TraditionalMuslim@reddthat.com 12 points 1 year ago

Appreciate the effort. Can we get a pinned post next time we have an update incoming and an ETA for how long the server will be down? Thanks.

[-] God_Is_Love@reddthat.com 9 points 1 year ago

No need to apologize, things happen and it's a learning curve! 🥰

[-] ticoombs@reddthat.com 10 points 1 year ago

When you have now... 500! active users and over 2.8k user accounts I feel somewhat responsible for people having a good experience. When have you used a service which lost data twice in a week! I'm annoyed in that these things are easily avoidable but they are causing issues. Moving forward I'll be putting in better more checks around deployments to make sure these cannot happen again.
I keep putting too much trust into the upgrade process when new versions come out expecting it to work without issues.

[-] burgundymyr@reddthat.com 16 points 1 year ago

I really appreciate the apology, and every web admin and sys admin has either made the same mistake, or watched someone else make it. I could absolutely see my younger self creating an instance of Lemmy and doing the exact same.

The fact that you acknowledged the mistake, learned from it, and communicated the improvements are all great signs that we're in the right place!

Thanks for all you're doing!

[-] ZagTheRaccoon@reddthat.com 7 points 1 year ago

Very sad, was having some great conversation that was wiped. Hope this doesn't happen too often but I understand.

[-] M1lt0n@reddthat.com 5 points 1 year ago

I was wondering why I couldn't login with my creds anymore... 🤣

[-] ticoombs@reddthat.com 8 points 1 year ago

I'm so sorry! But I'm super grateful you came back

this post was submitted on 12 Jul 2023
63 points (100.0% liked)

Reddthat Announcements

641 readers
1 users here now

Main Announcements related to Reddthat.

founded 1 year ago
MODERATORS