Lemmy World

There was an issue today with content from Lemmy World not federating to us. After pulling my hair out and testing the DubVee stack top to bottom, I got in touch with the awesome admins over at LW to try to figure out the problem. After they gave their federation service workers a good, stern talking to, we're now receiving content again.

Unfortunately, due to the amount of time (about 4 hours) the federation messages were getting stuck, there is a huge backlog that needs to catch up: about 90,000 messages. As of this writing, there are still 55,024 queued ActivityPub messages in flight (that includes backlog and current messages).

So just a heads up if you're not seeing posts/votes/comments from there or if they don't show up for a while. They will eventually arrive, though they'll be in the feed based on when they were published rather than when they arrive. On the bright side, they are receiving posts/votes/comments from us; you just might not get a reply right away.

Hopefully the queue works through overnight and things are back to normal tomorrow 🤞

Update: Backlog queue down to 37,150 now.

Update #2: 6,531 and falling. Looking like we'll be back in sync shortly.

Update #3: Backlog has finished syncing as of 23:55 EST.

mander.xyz

This one was my fault. lol. About 4 months ago, I was shoring up the firewall against some malicious traffic patterns, and mander.xyz 's server IP got caught in the CIDR block crossfire. Since I see content from their users frequently, I wasn't aware there was a problem. Turns out those were just relayed to us by the home instances of the communities. After I made a firewall allowance for mander's server, I had to reach out to their admin to reset the "last seen" date for DubVee in order to "bring us back to life" as their server had marked ours as dead.

So, all of the communities on mander.xyz should now look alive again. 🎉

My sincerest thanks to @rooki@lemmy.world, @ruud@lemmy.world, and @sal@mander.xyz for their assistance today.