cross-posted from: https://lemm.ee/post/52454753

Since the last update I've not been able to log into my fedia-account using interstellar. I keep getting the 500 internal server error. I can log in with my browser. Is fedia.io/mbin not supported anymore? As far as I know interstellar is/was the only app supporting mbin/fedia.

My apologies for the recent spate of problems. I think I’ve narrowed the problem down to the /m/fediverse and /m/random magazines. For some reason, mbin is generating an enormous amount of outbound delivery messages for these two magazines. I first tried removing the hashtags from /m/fedivese, but that was only a quick fix. So I deleted the magazine. (Note, the notifications appear to be related to the “microblog” function, and were originating from accounts on lots of mastodon instances, so I think there is a bug somewhere).

I noticed /m/random doing something similar. I have removed all the subscribers from that magazine to try to reduce the number of notifications it is sending. I don’t know if that will help - I have a feeling the instance can’t keep up with that happening in both random and fediverse.

Anyhow, the queues are draining fast now. I purged about 600000 queued delivery messages that (based on a random sample) all appeared to be associated with fediverse and random. That should let the rest of whatever is backed up get moving again. and hopefully stay moving.

This time it seems like federation to Fedia from elsewhere is working okay, but that federation from Fedia to elsewhere is not.

@jerry

Just noticed that my comments and posts from the last 24 hours don't seem to be appearing on outside instances at all.

Not seeing any new content for the last few hours, figured I would make a note of it.

The following instances will be offline briefly on Saturday, December 14 from 9am ET / 2pm UTC for approxmately 10 minutes: infosec.exchange infosec.town infosec.pub pixel.infosec.exchange books.infosec.exchange matrix/element.infosec.exchange relay.infosec.exchange meetup.infosec.exchange video.infosec.exchange infosec.press infosec.place fedia.io fedia.social elk,.infosec.exchange infosec.space convo.casa

The servers supporting these instances require a reboot. The Dell servers these instances run on take a very long time to boot, so I am estimating 10 minutes of downtime. It could be more, could be less.

We use live patches to minimize reboots needed for patching, however Ubuntu only provides livepatch support for a year, which is how long most of these systems have been running for.

This may be more of a client-side problem but the searching capabilities of mbin as a whole are so dreadfully bad. whenever i need to find a person or a community i first try fedia for good mesure yet most of the time it's not worth it as it doesn't show me what i want, afterwards i go into lemmy or mastodon find the thing i was looking for input it into the url and only then can i follow the person or community. And for apps like interstellar it's even worse since you can't input an url so you have to do it by logging into the browser version first. It's even harder to find people since for whatever reason magazines isn't showing them at all most of the time and the people option doesn't include a search option so your best bet is using global search for a chance that the specific person has been tagged in a post somewhere in the abise... Sorry for the gramatical errors btw. Oh and you can't follow hastags in mbin from what i have seen and tried...

We seem to be having trouble with a handful of the larger lemmy instances, as well as some misc other instances even though these instances aren't dead-dead: https://fedia.io/federation (at bottom of page - "Dead Instances").

I thought maybe it would've been a problem with a specific lemmy version, but there are several versions in the list and also some still-living Mastodon instances that are running reasonably up-to-date versions.

Jerry, I have a question why does fedia defederates Threads, but infosec.exchange doesn't? In the past fedia was federating with Threads, but recently they were suspended

It was my main place to follow Threads accounts, so it is kinda disappointing, but the inconsistent defederation just seems strange to me

You obviously don't own anyone anything, but I just would love to hear the reason/the explanation, was there something I missed? @Jerry @jerry@infosec.exchange

#fedia

Been getting this error page the last couple of days when creating a new thread, but the post is created:

Error 503 first byte timeout first byte timeout

Error 54113 Details: cache-fra-eddf8230069-FRA 1731425603 772093449

Varnish cache server

[Föderationstest] ab mbin 1.7.2 im #fediverse #fedia

It appears that sometime in the past few days, posts from people I've followed have stopped showing up in the feed. The latest from pluralistic @ mamot.fr for instance is not there right now. He's not blocked me, I can see it if I go to his profile. All that shows up is stuff posted to hashtags that I follow.

I guess it's a new bug?

Figured I'd drop this here in case fedia.io wants to check it out.

It’s been a long day. I will fix it when I am back in front of a computer. It might be a few hours. My apologies.

I am pretty sure the newest federated post is from 4 hours ago

Edit: Jerry is aware of the problem

I have sort of given up in fixing the problem, and will instead work on auto-detecting and auto-recovering when the problem happens.

I just saw this: https://every.to/p/the-disappearance-of-an-internet-domain

I have no idea if it's real, but if it is, that will be most unfortunate

After I resolved the federation issue, I had to clean up a few things and so the site may have been unavailable for a bit. I'm done fussing with it and will keep an eye on it to make sure things are working.

IF YOU SEE PROBLEMS - please let me know. As far as I know, I've fixed all of the federation and error 500 issues we've had, so please don't assume it's just more of the same if you see them.

Thanks for your patience.

Fedia.io is sort of like she Ship of Theseus right now - I literally replaced nearly everything trying to get it back working.

The problem ended up being a silent out of memory error that php-fpm was running into. I had to increase the memory limit to about 10x what the docs require to get it to work, but once I did that, it works great.

I was only able to sort this out after @bentigorlich recommended I move the site to debug mode (which requires me to lock everyone else out). Once I did that, it started giving some useful errors.

My apologies for the amount of time it took to fix this. I learned a lot about php today.

Hi all. As some of you have reported, outbound federation to at least some other instances is broken from fedia.io. At the moment. I don't know why and I don't have any leads as there are no logs or other indications of what is going wrong, but I am working on it.

This is a test

Hi all. Several of you have reported problems with fedia.io not federating with other instances correctly.

The cause is that rabbitmq crashed, but not all the way. It crashed to the point where new connections would timeout, but the service was still running such that it wouldn't auto restart. I will be creating some automation to detect that proactively and restart rabbitmq if/when it happens again.

How user accounts work with the fediverse?? Can I sign in with my Fedia account on other Lemmy, Mbin etc instances?

#fedia

for two days, threads posted on fedia.io magazines apparently do not show up on lemmy > this does not only affect the poetry magazine, but also https://fedia.io/m/FloatingIsFun - i guess the same applies to other fedia.io magazines > i have already checked both lemmy.world and infosec.pub: no new threads show up

We made some changes a few minute ago that we hope fixes the problem. There may be some other lingering issues, but I am hoping the voting problem is fixed now. Let me know below if you continue to see that problem.