this post was submitted on 18 Jun 2024
32 points (100.0% liked)

Mander

491 readers
3 users here now

founded 3 years ago
MODERATORS
Sal@mander.xyz
32
Version update and server upgrade! (mander.xyz)
submitted 10 months ago* (last edited 10 months ago) by Sal@mander.xyz to c/mander@mander.xyz
51 comments fedilink hide all child comments
 

The instance has been updated to version 0.19.4!

You can see the official release post and notes here: https://lemmy.ml/post/16585359?scrollToComments=true

Since this update required some manual intervention anyway to update the database version, I took the opportunity to perform some overdue server maintenance and upgrades.

I have migrated the server to a more powerful configuration, and changed the way image files are being stored and served (object storage as opposed to filesystem storage).

I am also aware of some federation issues that have affected this and other instances. I am not sure whether this upgrade will deal with these problems automatically. I will be looking into it and hopefully will implement some solution soon.

Sorry for the downtime experienced in the past few days, but hopefully with the more powerful server the experience will be smoother and will remain so for a while.

For other admins and users interested, this is a bit of information about the server

The previous configuration:

Hosting provider: serverspace.io

8 GB RAM

4 Core CPU

50 Mbps network

400 GB SSD

Cost: $55 / month

The 400 GB SSD was being used to store both the server's database and the image files, and this component in isolation had a monthly cost of $36. I have migrated the images to an object storage provider ($6/month for 500 GB), and used the resources previously used for disk space to increase the other parameters. The new configuration is a lot better and less expensive.

New configuration:

Hosting provider: serverspace.io

Object storage provider: contabo.com

16 GB RAM

6 core CPU

50 Mbps network

100 GB SSD

500 GB Object storage

Cost: $47/month

Thanks a lot to the lemmy and pict-rs devs for the hard work they put into developing the infrastructure!

top 50 comments
sorted by: hot top controversial new old
[–] Shdwdrgn@mander.xyz 6 points 10 months ago (1 children)

Wow the new server is just flying along!

[–] Sal@mander.xyz 5 points 10 months ago

Good to hear!

[–] BlastboomStrice@mander.xyz 5 points 10 months ago

Lets gooooo🔥🔥🔥

Well done!

[–] Blaze@reddthat.com 5 points 10 months ago (1 children)

Well done!

[–] Sal@mander.xyz 6 points 10 months ago (10 children)

Thanks!

Since the upgrade, I can see (here) that activities from lemmy.world are being fetched significantly faster. I will let it run for some time and hopefully this speed-up is enough to catch up.

[–] Blaze@reddthat.com 3 points 10 months ago

Great, looks promising!

You seem also to be catching up with the backlog somehow? https://grafana.lem.rocks/d/cdfzs0dwal3pca/federation-health-time-behind?orgId=1&var-instance=All&var-remote_instance=mander.xyz&var-remote_software=All&from=now-12h&to=now

Hopefully the trend will continue, let's wait a few hours and see

[–] Bl4ze@lemmy.world 1 points 10 months ago (1 children)

Test from a LW account to see how long it takes to show up (I posted at 18:24 GMT+1)

[–] Sal@mander.xyz 1 points 10 months ago

Well it is here now! Almost caught up :-)

[–] M0oP0o@lemmy.world 1 points 10 months ago

Yay, down to one week delay. Looking forward to catching up.

load more comments (7 replies)
[–] ratel@mander.xyz 3 points 10 months ago

Well done and many thanks!

[–] lil@lemy.lol 3 points 10 months ago (2 children)

Now update to version 0.19.5, please

[–] Blaze@reddthat.com 4 points 10 months ago

I hope that's a joke, give this poor admin some rest!

[–] Sal@mander.xyz 3 points 10 months ago (1 children)

As soon as it comes!

[–] Blaze@reddthat.com 3 points 10 months ago (1 children)
[–] Sal@mander.xyz 2 points 10 months ago (1 children)

Aaaah, thanks! 😆 It is done now.

[–] Blaze@reddthat.com 2 points 10 months ago (1 children)

Already? Impressive!

[–] Sal@mander.xyz 3 points 10 months ago

This was a small update so it only required me to get to my computer. I just had to change 0.19.4 to 0.19.5 and restart the docker containers :)

The 0.19.3 -> 0.19.4 was lot more involved because of the database upgrade and all of the extra server stuff that I needed to catch up on.

[–] lvxferre@mander.xyz 2 points 10 months ago* (last edited 10 months ago) (3 children)

Sal, I got some error here: couldnt_get_posts. That appears every time that I try to visualise the instance through the "Subscribed" communities.

The error does not appear if I browse the instance through "Local" or "All", nor if I check the list of communities that I'm subscribed to. The error happens across devices and front-ends (tested with Linux Firefox, Android Firefox, Jerboa).

EDIT/UPDATE: I unsubscribed manually to a few communities each time, checking if the front page loaded fine. Eventually, it did.

Re-subscribing to either !technology@lemmy.world or !fediverse@lemmy.world seems to trigger the error again.

[–] Sal@mander.xyz 6 points 10 months ago (3 children)

(@Shdwdrgn@mander.xyz ) I think it is related with the processing of so many activities per second while syncing with Lemmy.World.

I have tried changing some rate limits and the number of maximum database connections, but I get similar errors while moving around the site. The server is running well below 100% on all metrics, and the logs don't produce an obvious error. I will wait until we catch up with Lemmy.World and see if that fixes the issue. If it doesn't, then I will trouble shoot more in-depth.

[–] acockworkorange@mander.xyz 3 points 10 months ago (1 children)

Letting you know I no longer see the issue. Thanks, Sal!

[–] Sal@mander.xyz 2 points 10 months ago

Great to hear! Thanks!

[–] lvxferre@mander.xyz 3 points 10 months ago

Thank you!

[–] RobotToaster@mander.xyz 1 points 10 months ago (2 children)

There's some potential solutions to this mentioned here https://github.com/LemmyNet/lemmy/issues/4277

[–] Sal@mander.xyz 3 points 10 months ago (1 children)

Thanks! The database is currently fetching lemmy world data as fast as possible, so this might be saturating the database. Once the syncronization is tested, I will try to tune postgres if it still happens.

[–] lvxferre@mander.xyz 3 points 10 months ago (1 children)

The issue seems to be gone for me. And the instance "feels" considerably faster than it did before :-D

[–] Sal@mander.xyz 3 points 10 months ago

Great, thanks for letting me know!! I did a bit of research and I think that the next time I need to upgrade the instance I will move it to a dedicated server instead of a virtual private server. But hopefully it runs smoothly for some time! :)

[–] Sal@mander.xyz 2 points 10 months ago (2 children)

Hey @RobotToaster@mander.xyz , @lvxferre@lemmy.ml , and @Shdwdrgn@mander.xyz - we are now in sync with lemmy.world and the database under much less stress. This seems to have fixed the problems that I observed on my end. Are your problems also resolved?

[–] RobotToaster@mander.xyz 3 points 10 months ago

I just resubbed to the communities that were causing issues and it looks like everything is working well now, thanks.

[–] Shdwdrgn@mander.xyz 2 points 10 months ago (1 children)

Ah! It looks like it JUST fixed it, I had checked about 15 minutes ago and still couldn't get my subscribed list to load, but it does appears to be loading up now. I'll reset my preferences and keep an eye on it through the day -- hoping for the best!

[–] Sal@mander.xyz 1 points 10 months ago (1 children)

It should have been fixed about 13 minutes ago! Thanks for checking! If problems persist I will try to fine-tune the database settings as suggested above.

[–] Shdwdrgn@mander.xyz 3 points 10 months ago (1 children)

There was a brief outage soon after I left that last reply, assume you were doing another round of maintenance at the time, but otherwise it's still looking good from here.

[–] Sal@mander.xyz 2 points 10 months ago

Ah, thanks. I think that was when I reboot the instance to change a setting that controls how post thumbnails are generated, as the new setting was failing to generate thumbnails for many posts.

[–] Shdwdrgn@mander.xyz 4 points 10 months ago* (last edited 10 months ago) (2 children)

That's the same error I was asking about below, and wondering if it had to do with the current status of collecting all the content. If so, it sounds like it could be another day or two before we can view our front pages again?

[–] lvxferre@mander.xyz 2 points 10 months ago (2 children)

Sorry for the double reply.

Check if you're subscribed to any of those communities: !technology@lemmy.world , !comicstrips@lemmy.world or !memes@lemmy.world. If you are, try to unsubscribe and see if your front page works again. For me it did the trick.

!fediverse@lemmy.world was also giving me problems, but apparently it's fine now, so odds are that the problem will eventually go away on its own.

[–] Shdwdrgn@mander.xyz 2 points 10 months ago

All of this sounds like the issue is related to the new server still loading up the communities. I'm subbed to a pretty good list so it's probably best just to wait for the server to catch up. Thanks for the info though!

[–] acockworkorange@mander.xyz 2 points 10 months ago

Unsubscribing from comic strips did the trick a couple days ago, thanks! And just now I’ve subscribed back and everything is normal. So feel free to subscribe again.

[–] lvxferre@mander.xyz 1 points 10 months ago

For now, one option is to temp change your settings to browse by "Local" or "All". Or alternatively temporarily unsubscribe off a few comms; if you're subscribed to a small number of comms, some trial and error might help you to find which ones are causing the issue. (For me it was two LW comms).

[–] RobotToaster@mander.xyz 2 points 10 months ago

Having the same issue here unfortunately.

[–] Bl4ze@lemmy.world 2 points 10 months ago (2 children)

New test, 23:01

[–] Sal@mander.xyz 2 points 10 months ago (1 children)

It works now :)

[–] Blaze@reddthat.com 2 points 10 months ago (1 children)

It does, that's great! Well done!

[–] Sal@mander.xyz 2 points 10 months ago

Thanks! All seems to be in order now 😌

[–] Blaze@reddthat.com 1 points 10 months ago

Great!

[–] fossilesque@mander.xyz 1 points 10 months ago

🎉 🌞

load more comments
view more: next ›