this post was submitted on 12 Jun 2023

390 points (99.7% liked)

Lemmy.World Announcements

30938 readers

2 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

DM https://lemmy.world/u/lwreport
Email report@lemmy.world (PGP Supported)

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Join the team

founded 2 years ago

MODERATORS

ruud@lemmy.world

lwadmin@lemmy.world

lwCET@lemmy.world

jelloeater85@lemmy.world

Serinus@lemmy.world

lw_mod_notification@lemmy.world

390

How is lemmyworld so stable? (monero.house)

submitted 2 years ago by maltfield@monero.house to c/lemmyworld@lemmy.world

129 comments fedilink hide all child comments

At the time of writing, Lemmyworld has the second highest number of active users (compared to all lemmy instances)

https://lemmy.fediverse.observer/list

Also at the time of writing, Lemmyworld has >99% uptime.

By comparison, other lemmy instances with as many users as Lemmyworld keep going down.

What optimizations has Lemmyworld made to their hosting configuration that has made it more resilient than other instances' hosting configurations?

https://lemmy.ml/post/1221008

top 50 comments

sorted by: hot top controversial new old

[–] ruud@lemmy.world 3 points 2 years ago (20 children)

Thank you for all the compliments.

This ride reminds me of Mastodon.world in November. Details on that are here: https://blog.mastodon.world/and-then-november-happened

So I started lemmy.world on a 2CPU/4GB VPS. Keeping an eye on the performance. Soon I decided to double that. And after the first few thousand of users joined, doubled it again to 8CPU/16GB. That also was the max I could for that VPS type.

But, already I saw some donations come in, without really asking. That reminded me of the willingness to donate on Mastodon, which allowed me to easily pay for a very powerful server for mastodon.world, one of the reasons it grew so fast. Other (large) servers crashed and closed registrations, I (mainly) didn't.

So, I decided to buy the same large server (32cpu/64threads with 128GB RAM) as for masto (but that masto one has double the RAM). With the post announcing that, I also mentioned the donation possibilities. That brought a lot of donations immediately, already funding this server for at least 2 months. (To the anonymous person donating $100 : wow!).

Now next: to solve the issue with post slowness. That's probably a database issue.

And again: migration took 4 minutes downtime, and that could have been less if I wasn't eating pizza at the same time. So if any server wants to migrate: please do! If you have the userbase, you'll get the donations for it. Contact me if you have questions.

[–] Soullioness@lemmy.world 1 points 2 years ago (3 children)

What does it cost per month to operate your servers, namely this one?

load more comments (3 replies)

[–] Xaphanos@lemmy.world 1 points 2 years ago (1 children)

As someone "in the business", but not nearly as technical as you... How far can a single instance scale? Can a load balancer spread it over mulitple front-ends to handle user load? Can the back-end be re-worked to handle hundreds of millions of user operations per second? Can it work with a CDN? Can a single "Lemmy.World" site exist as a distributed site - with hundreds of servers spread across dozens of sites across the globe?

I expect this entire line of thought is antithetical to the entire Lemmy philosophy of distributed operation. I expect that the "correct" way is to spin off "NA.Lemmy.World", EMEA. Lemmy.World", APAC.Lemmy.World", etc. as separate servers. Is that correct?

Thanks.

load more comments (1 replies)

[–] million@lemmy.world 1 points 2 years ago (1 children)

Nice job, thanks very much for the write up.

Out of curiosity are you cloud hosting or do you own a server on a rack somewhere? Scaling with Kubernetes or VMs or just running bare-metal?

load more comments (1 replies)

[–] Dream_state@lemmy.world 0 points 2 years ago (2 children)

Am I able to use the same account to login to mastodon.world? Or do I need to make an account there too? Never used mastodon but vining the fediverse stuff

load more comments (2 replies)

[–] grouvie@lemmy.help 0 points 2 years ago

Interesting writeup. I'm curious about the resource usage of the Lemmy backend and frontend deployments. Do you have any insights on the resource utilization of these deployments?

[–] taj@lemmy.world 0 points 2 years ago

Fantastic. Thanks for all you've done, and are doing!!

[–] SteelBeard@lemmy.world 0 points 2 years ago

Hey, you rock. This place is pretty cool.

load more comments (13 replies)

[–] neighbourbehaviour@lemmy.world 2 points 2 years ago* (last edited 2 years ago) (1 children)

It's known in the industry as the throw-hardware-at-it optimization. It's often effective and what's needed to buy time for software optimization to come in.

[–] OsrsNeedsF2P@lemmy.ml 1 points 2 years ago (1 children)

As someone who got burnt out on one of their last businesses due to optimizing too early - Yes!!!

Doing it "properly" with "stateless servers" and "autoscaling" with "Kubernetes" costs a hell of a lot more money than a 64 Core server with 256 GB of RAM

load more comments (1 replies)

[–] psychothumbs@lemmy.world 1 points 2 years ago (6 children)

Looks like the guy who runs it runs a lot of fediverse servers, I guess he knows what he's doing: https://lemmy.world/u/ruud

[–] kiwifoxtrot@lemmy.world 2 points 2 years ago

@ruud runs a top 10 mastodon server.

load more comments (5 replies)

[–] andrew@radiation.party 1 points 2 years ago (7 children)

Likely experience and knowledge improving the quality of deployment. Most instances are likely underspecced, are on hosts that aren't easy to scale up with, or are maxed out in their current offering tier (lemmy.ml comes to mind there)

load more comments (7 replies)

[–] wilberfan@lemmy.world 0 points 2 years ago

Just subscribed at Patreon to support the cause! 👍

[–] netburnr@lemmy.world 0 points 2 years ago* (last edited 2 years ago)

He has been posting updates along the way. It's a combo of upgrading the server as it hits its limits and tuning of his web proxy and docker container to handle the increased load and federation requirements.

Been doing an amazing job of it too. I just randomly chose this instance and I'm glad.

Edit. Here is his last post https://lemmy.world/post/75556

[–] PriorProject@lemmy.world 0 points 2 years ago (15 children)

I'm not an admin, but have followed the sizing discussions around the lemmyverse as closely as I can from my position of lacking first-hand knowledge:

lemmy.ml is the biggest instance by user count, but runs on incredibly modest 8-cpu hardware. Their cloud provider doesn't provide any easy scale up options for them, so they can't trivially restart on a bigger VM with their db and disk in place. I suspect this means that instance is going to suffer for a bit as they figure out what to do next.
lemmy.world on the other hand was running on a box at least twice as big as lemmy.ml at last count, and I believe they can go quite a bit bigger if they need to.
The lemmy.world admins also run mastodon.world and lived through the twitterpocalypse, seeing peak user registrations rates of 4k per hour. So this is not their first rodeo in terms of explosive growth, I'm sure that experience gives them some tricks up their sleeve.
The admin team is pretty clearly technically strong. If I recall correctly, ruud is a professional database admin. One of the spooky parts of Lemmy performance-wise is the db. If ruud or others on the admin team custom-tuned their pg setup based on their own analysis of how/why it's slow, they may be getting more performance per CPU cycle than other instances running more stock configs or that are cargo-culting tweaks that aren't optimal for their setup without understanding what makes them work.

I'm surprised that sh.itjust.works isn't growing faster. They also have a hefty hardware setup and seemingly the technical admins to handle big user counts. I wonder if it's a branding problem, where lemmy.world sounds inviting and plausibly serious where sh.itjust.works sounds like clowntown even though it's run by a capable and serious team.

[–] RetroEvolute@lemmy.world 2 points 2 years ago* (last edited 2 years ago) (1 children)

I originally signed up with sh.itjust.works, but I wanted to be on the instance with the majority of migrants.

Also, it sounds dumb, but I think the sh.itjust.works domain is just kinda weird, technically has a "curse word" in it (not that I personally care), and they don't support NSFW content (which isn't just used for porn). So, it didn't make sense to have that as my home instance. 🤷‍♂️

Edit: Also, this is my first comment on here! Hello world! 👋

[–] PriorProject@lemmy.world 2 points 2 years ago (1 children)

Yeah, I get it. Naming optics aside, it seems an instance with a lot of headroom relative to others, with a capable team. Would be near the top of my word-of-mouth options in spite of the idiosyncratic name.

[–] liontigerwings@sh.itjust.works 1 points 2 years ago

It's been running a little slow today though so maybe not as much headroom as you think

[–] WaffleFriends@lemmy.world 1 points 2 years ago

I had a very similar thought process when choosing my instance. lemmy.world seemed like it would be more open to new users than an instance named sh.itjust.works. Idk why that was my thought process but I’m here now

[–] Master@lemmy.world 1 points 2 years ago (3 children)

Can confirm... I didnt sign up for sh.itjust.works solely because of the name... I dont particularly want that attached to every post I make.

load more comments (3 replies)

[–] StrayPizza@lemmy.world 1 points 2 years ago

I hope lemmy.ml can upgrade at some point. A lot of the slowness I'm running into is trying to browse/discovery communities that happen to live on that instance.

[–] darkwing_duck@sh.itjust.works 1 points 2 years ago* (last edited 2 years ago)

That's actually awesome for users of sh.itjust.works. Like myself.

[–] Pspspspspsps@lemmy.world 1 points 2 years ago* (last edited 2 years ago) (5 children)

I wonder if it's a branding problem, where lemmy.world sounds inviting and plausibly serious where sh.itjust.works sounds like clowntown

That was my thought process when choosing an instance tbh. I'm not a tech person, I looked at the list and lemmy.world was the first 'safest feeling' instance that had open sign up. I saw sh.itjust.works and didn't even check their sign up process, there was too many periods in the strange name and it just looks weird to me as someone not used to these things. Edit: spelling

[–] s4if@lemmy.world 2 points 2 years ago

nah, I'm bit regretting not signing up on their instance. sh.itjust.works is a cool name and can be a brag point. lol. lemmy world is a bit too generalist, but I won't migrate there as ruud (the admin of lemmyworld) is doing a good job managing the instance. I appreciate that. :)

[–] Z______@lemmy.world 0 points 2 years ago* (last edited 2 years ago) (1 children)

I definitely second the motion on it being a branding problem. Stuff like sh.itjust.works seem to me like something that dark basement tech nerds would come up with that is "edgy" and really only used by them and other people like them.

I'm not really into the ironic "edgy" aesthetic and part of the struggle with this transition for me has been orienting myself in the space because I don't want to commit to some "sketchy" edgelord URL

[–] darkwing_duck@sh.itjust.works 1 points 2 years ago (2 children)

something that dark basement tech nerds would come up with that is “edgy” and really only used by them and other people like them.

That's exactly what it is and why I love it. The whole thing about this federated networking is that it doesn't matter where you signed up.

[–] ericjmorey@lemmy.world 1 points 2 years ago (3 children)

Where you sign up entirely determines your local feed.

[–] darkwing_duck@sh.itjust.works 1 points 2 years ago

Just like with reddit, I don't use defaults.

load more comments (2 replies)

load more comments (1 replies)

load more comments (3 replies)

[–] wheen@lemmy.world 1 points 2 years ago (1 children)

Can none of this scale horizontally? Every mention of scaling has been just "throw a bigger computer at it".

We're already running into issues with the bigger servers being unable to handle the load. Spinning up entirely new instances technically works, but is an awful user experience and seems like it could be exploited.

load more comments (1 replies)

[–] Druidgrove@lemmy.world 1 points 2 years ago (3 children)

I'm now going to start incorporating "Sounds like clowntown" into my everyday conversations - that's funny!

load more comments (3 replies)

[–] maltfield@monero.house 0 points 2 years ago (1 children)

Right, but if you don't have a cache setup, then the DB gets taxed. At a certain point a cache looses its benefit, but an enormous amount of savings can be made (to backend DB calls, for example) by just caching all API reads for ~60 seconds.

[–] andrew@radiation.party 0 points 2 years ago* (last edited 2 years ago) (1 children)

Ensuring there's no data leakage in those cached calls can be tricky, especially if any api calls return anything sensitive (login tokens, authentication information, etc) but I can see caching all read-only endpoints that return the same data regardless of permissions for a second or two being helpful for the larger servers.

It's also worth noting that postgres does its own query-level caching, quite aggressively too. I've worked in some places where we had to add a SELECT RANDOM() to a query to ensure it was pulling the latest data.

[–] maltfield@monero.house 0 points 2 years ago (1 children)

In my experience, the best benefits gained from caching are done before the backend and are stored in RAM, so the query never even reaches those services at all. I've used varnish for this (which is also what the big CDN providers use). In Lemmy, I imagine that would be the ngnix proxy that sits in-front of the backend.

[–] PriorProject@lemmy.world 0 points 2 years ago (7 children)

I haven't heard admins discussing web-proxy caching, which may have something to do with the fact that the Lemmy API is currently pretty much entirely over websockets. I'm not an expert in web-sockets, and I don't want to say that websockets API responses absolutely can't be cached... but it's not like caching a restful API. They are working on moving away from websockets, btw... but it's not there yet.

The comments from Lemmy devs in https://github.com/LemmyNet/lemmy/issues/2877 make me think that there's a lot of database query optimization low-hanging fruit to be had, and that admins are frequently focusing on app configs like worker counts and db configs to maximize the effectiveness of db-level caches, indexes, and other optimizations.

Which isn't to say there aren't gains in the direction your suggesting, but I haven't seen evidence that anyone's secret sauce is in effective web-proxy caches.

[–] maltfield@monero.house 0 points 2 years ago (1 children)

Yeah, that's exactly why I'm asking this question. All the effort seems to be going into the DB -- but you can have a horribly shitty DB and backend but still have a massively performant webserver by just caching away the reads to RAM.

I didn't see any tickets about this on the GitHub, which is why I'm asking around to see if there's actually some very low-hanging-fruit for improving all the instances with a frontend RAM cache.

load more comments (1 replies)

load more comments (6 replies)

[–] isosphere@beehaw.org 0 points 2 years ago* (last edited 2 years ago)

sh.itjust.works

on paper i'd be on this instance but the name is quite terrible and gives me little confidence in the administration

load more comments (5 replies)

load more comments