Shortly after upgrading to Mbin 1.7.1-rc1, php ran out of workers. I dramatically increased the limit. It isn’t clear to me why that happened and if it’s related to the upgrade or just coincidental. My intuition is that it’s related, but I have no evidence.

Hello everyone. I just upgraded fedia.io to mbin 1.7.1-rc1. One of the notable changes is that mbin is deprecating mercure, which is the component that provided streaming updates. As such, you will have to refresh the web page to see new posts and comments.

The (relatively new) server that Fedia.io was running on, a Hetzner AX 162-R, died overnight. Hetzner tells me that the main board failed and had to be replaced. In the process of repairing, the raid set got corrupted and would no longer boot.

Every single AX 162 (R or M) I’ve rented from Hetzner has failed now at least once. This was the last one I had. It was on my to do list to move fedia.io to a Dell server with the same specs. I knew this was going to happen, but I didn’t get it done in time.

For those of you who have been following along, Fedia has been cursed from the beginning. The kbin software was a god damned disaster, and very fortunately the mbin team spent an incredible amount of time and patience to help me sort out the many problems, nearly all of which are fixed now.

Except for the random occurrences where federation breaks due to an as-yet-unknown bug, the main stability issue has been hardware. I have had excellent luck with Hetzner’s Dell servers, so I am hopeful that is now fixed as well. The challenge is that the Dell server is quite expensive ($350 per month) so I will be looking to find a more cost effective way to host fedia.io, given the very small number of active users.

I will be rehoming fedia.io to a less expensive server the afternoon of July 1 - exact timing is TBD. Downtime should last about 2 hours. The current server is quite expensive and donations are dwindling, which is normally ok, but I am losing my job and have to be a bit more frugal.

Yesterday, the fedia.io server locked up. I was able to reboot it remotely and it came up clean. After less than an hour, the server froze again. This happened several more times throughout the day. Unfortunately, there were no logs recording what happened, and nothing on the console - just frozen hardware.

I contacted Hetzner early this morning and they diagnosed the server as having a faulty motherboard. Hetzner replaced the board and rebooted the server, and so far the server has been stable.

I have had pretty bad luck with this particular model of server from Hetzner, so I do not have confidence that this won't happen again, and so will be looking to migrate to a different type of server that is hopefully more stable and less expensive (I am losing my job at the end of June, and so need to save all the cash I can).

Fedia.io had a few issues over the past 24 hours - sometimes working find till you click on certain posts, which result in an error 500, and other times just getting an error 500 no matter what.

The first issue I found is that amqproxy, which helps to reduce the load on the server between the queue runners that process incoming and outgoing posts and rabbitmq. I found this morning that amqproxy was consistently failing, despite there being no apparent problem. I bypassed amqproxy, since the server can handle the load fine without amqproxy. That seemed to work and things returned to normal. A few hours later, the site started responding with error 500 to nearly all requests. This happened because the database server ran our of connections. The 300 it was set to should have been plenty, but clearly it was not. I've set that to 3000 and so far, so good.

My apologies for the instability. I continue to learn the nuances here and will keep making the service more reliable as I go.

The server fedia.io had been running on started developing stability problems overnight from Thursday April 4 to Friday April 5. By Saturday (today), the system was completely unbootable. After attempting to resolve the hardware issue with Hetzner (the ISP) for about 6 hours, I gave up and moved the site to a new server. All that took quite a lot of time, during which fedia.io was not available. My apologies for this.

Fedia.io is now running on it's own stand-alone server. This server is costing me $210/month, so any contributions are welcome. Please let me know of any issues.

I am going to be rehoming fedia.io to a new server. Today, fedia.io runs on shared app server and shared database server. I will be moving fedia.io to a dedicated 48 core/96 thread AMD epyc server with 256GB of RAM, which will house both the app and the database. This is less about fedia.io and more about the other apps running on the existing servers are that suffering due to performance issues with the mbin application, particularly the database queries. (Note: this isn't a slam on mbin - it just needs more optimization, plus fedia.io is a pretty busy server).

You have no doubt noticed that federation is breaking again. I am painfully aware of it. The issue is with the symphony queue runner that processes incoming messages from other instances. Occasionally, the server receives a message that causes the queue runner to die. I have to manually remove the offending message out of rabbitmq. The message does not appear to be malicious, rather there is something malformed in an otherwise legit looking post that causes the queue to die. I am working with the mbin team to track down what it is about the messages that causes the problem, but sadly until I there is a fix, this is going to keep happening