this post was submitted on 04 Jun 2023

373 points (97.7% liked)

Lemmy

14183 readers

5 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 5 years ago

MODERATORS

nutomic@lemmy.ml

373

lemmy.ml is overloaded, use other instances instead (lemmy.ml)

submitted 2 years ago* (last edited 2 years ago) by nutomic@lemmy.ml to c/lemmy@lemmy.ml

349 comments fedilink hide all child comments

This site is currently struggling to handle the amount of new users. I have already upgraded the server, but it will go down regardless if half of Reddit tries to join.

However Lemmy is federated software, meaning you can interact seamlessly with communities on other instances like beehaw.org or lemmy.one. The documentation explains in more detail how this works. Use the instance list to find one where you can register. Then use the Community Browser to find interesting communities. Paste the community url into the search field to follow it.

You can help other Reddit refugees by inviting them to the same Lemmy instance where you joined. This way we can spread the load across many different servers. And users with similar interests will end up together on the same instances. Others on the same instance can also automatically see posts from all the communities that you follow.

Edit: If you moderate a large subreddit, do not link your users directly to lemmy.ml in your announcements. That way the server will only go down sooner.

top 50 comments

sorted by: hot top controversial new old

[–] anji@lemmy.anji.nl 3 points 2 years ago (10 children)

Sadly, I feel like the Fediverse, based on ActivityPub, was fundamentally designed wrong for scaling potential. I do like Fedi and I like ActivityPub, but I think instances should not have to be responsible for all of this:

Owning user accounts
Exclusively host communities
Serving local and remote users webpages and media
Never going down, as this results in users and content becoming unavailable

Because servers "own" the user accounts and communities it's not trivial for users to switch to a different instance, and as instances scale their costs go up slightly exponentially.

I wish the Fediverse from the beginning was a truly distributed content replication platform, usenet-style or Matrix-style, and every instance would add additional capacity to the network instead of hosting specific communities or users.

I guess it's a bit too late for a redesign now... Perhaps decentralized identifiers will take us there in some form in the future.

[–] gnoop@lemmy.ml 2 points 2 years ago

While it might not be too late for that update, it would require some reconciliation to happen. There's the potential for multiple users and communities of the same name across servers that would need to be considered.

load more comments (9 replies)

[–] bilb@lemmy.ml 2 points 2 years ago (3 children)

I'm going to set up a general purpose instance tomorrow with the intention of handling a relatively large number of users. The main problem is choosing a domain!

[–] Nicarlo@lemmy.ca 2 points 2 years ago (5 children)

I was also contemplating setting up a new instance for this. I have 100s of gigs of unused ram, CPUs on idle and a 10gbit connection looking for something to do. The only issue I couldn't figure out was the name. I own itjust.works was thinking of something clever subdomain to use with it. I'm glad I'm not the only one with this issue

[–] autisticaudioguy@lemmy.ml 3 points 2 years ago* (last edited 2 years ago) (2 children)

sh.itjust.works

[–] Nicarlo@lemmy.ca 2 points 2 years ago (3 children)

I did it! https://sh.itjust.works

Credits go to you for the naming

[–] seirim@lemmy.ml 1 points 2 years ago (1 children)

Dude killer url, nice one! Question for all, I clicked their link and went there and it’s an instance, surely. I tried to comment on their post, but was required to sign in.. I’m already signed in over here, I gotta sign in there, too? Anyhow I tried to sign in with my lemmy.ml creds but that didn’t work. How can I interact with posts there?

[–] MBM@lemmy.world 1 points 2 years ago

When you open a post it should show you instructions on the right: use the search bar in lemmy.ml to search for [!main@sh.itjust.works](/c/main@sh.itjust.works)

load more comments (2 replies)

load more comments (1 replies)

load more comments (4 replies)

[–] TerrorBite@meow.social 1 points 2 years ago (7 children)

Naming things is one of the two most difficult issues in IT, alongside cache validation and off-by-one errors.

[–] snowe@lemmy.ml 1 points 2 years ago

choosing the name for my instance was easy. programming related? programming.dev it is!

[–] MyNameIsIgglePiggle@lemmy.ml 1 points 2 years ago (1 children)

I'm getting the following error reading this post: "item at index 2 does not exist"

Should I post this on stack overflow or some other Lemmy help community?

load more comments (1 replies)

load more comments (5 replies)

[–] bilb@lemmy.ml 1 points 2 years ago

It's a week later, but I did get this done finally. I've set up https://lem.monster/ . Still doing some tweaking, but it's open.

[–] 777@lemmy.ml 1 points 2 years ago (1 children)

I don't know what happened but in the last half hour the website has become highly responsive again. Thank you admins for your hard work.

load more comments (1 replies)

[–] ionhowto@lemmy.ml 1 points 2 years ago (3 children)

I use kbin too.

New to this feedverse or how you call it.

Why isn't there one login that can post on all platforms and I have to signup on each separately?

If there is, you're not making it obvious I guess.

[–] anders@rytter.me 1 points 2 years ago (1 children)

@nutomic @ionhowto you dont have to sign up on multiple instances. if you want to comment a post on another instance, copy the url and paste in into the search field and then your current instance will fetch the post so you can comment on it.

load more comments (1 replies)

load more comments (2 replies)

[–] zeerooth@lemmy.antemeridiem.xyz 1 points 2 years ago

I've made https://lemmy.antemeridiem.xyz/ to help take off some of that load. New registrations are welcomed and it should be maintained for a very very long time 🎂

[–] anders@rytter.me 1 points 2 years ago

@nutomic lemmy.world is a new instance which can also be used.

[–] spaghetti_carbanana@krabb.org 1 points 2 years ago

I hope it's not inappropriate to comment this here, but if anyone's looking for another space to join, I'm in the process of building Krab Borg. It would be lovely to have people to help fill it out and diversify the communities, as well as suggest what the local ones should look like as I have no idea.

I'm trying to balance not reinventing the wheel/duplicating existing communities 100 times but also still supporting the idea of decentralisation and creating some duplicates (though this isn't hard and fast, I'm open to feedback).

I've seeded it with some communities from other servers (including a bunch from lemmy.ml) to get things moving a bit as well.

[–] aksdb@feddit.de 1 points 2 years ago (2 children)

I think lemmy will be bitten in the ass by not having considered clustering/horizontal scaling from the start. Federation alone as a scaling mechanism is only feasible for "nerds". But if the network wants to grow, we will need a few scale-able large hosted instances. And if their only choice is to scale vertically, there will be a hard limit (unless we put a good old Mainframe somewhere ^^).

Another downside of this design is: you can't run it with high availability. If there's only one process per instance, updating it will mean the whole instance is down. Sure, if all goes well this downtime is under a second. But if it doesn't go well or if a migration is needed, this might quickly become hours.

[–] federico3@lemmy.ml 1 points 2 years ago (1 children)

Indeed. If a big instance like lemmy.ml was to be shut down all the communities would be lost. This is simply not sustainable. Why would users put effort building a community if it could be gone at any time?

[–] aksdb@feddit.de 1 points 2 years ago (11 children)

That however would be a different problem. A horizontally scaled instance would be able to cope with more users, but if it shuts down for monetary, personal, or whatever reason, it's still down.

Protecting a community from this is what the decentralized part is for. That is already in place.

(Although there is a middle ground where you could design the system in a way that one instance is mirrored and load-balanced across different hosters. That would actually also be quite interesting to have. But that's another layer of complexity on top.)

load more comments (11 replies)

[–] PriorProject@lemmy.world 1 points 2 years ago (1 children)

I think you probably underestimate how far one can get with "vertical" scaling. Here's the dockerfile: https://raw.githubusercontent.com/LemmyNet/lemmy/release/v0.17/docker/prod/docker-compose.yml

It includes 4 different containers... so there's a way to scale out to 4 machines right away. Maybe not every container is doing an equal amount of work... but there's some amount of immediately available machine-splitting.
I'm no expert, but I believe that at least the lemmy and lemmy-ui containers are stateless. If so, they're horizontally scalable already.
Postgres then would likely be the main bottleneck. But postgres offers read-replicas, so again the write-load and the read-load can be hosted on separate machines. And if there's enough read-load, you can have many replicas.

Other comments from the admins have shown that lemmy.ml today is running on a single eight-core box and it's currently hosting 30k registered users and over 1k active. So how much more compute capacity can we throw at "vertical" scaling on the current software architecture?

Just by going to a bigger single box, we can get 128 cores with no problem, a 16x bump in capacity. Does that get us to at least to 300k registered + 10k active?
Splitting the containers onto 4 separate machines. Does that get us 2x more?
Adding PG read-replicas and additional lemmy/lemm-ui containers would allow us to expand our instance footprint to maybe 6 physical machines should get us another 2x or more in performance.

Conservatively, that's 100x the computing capacity of the current hardware and could potentially support 1m registered users and 50k active. Now, I don't REALLY expect this to be possible today, there will be many software bottlenecks found along the way to scaling a single instance this large. But my point is that there's already a medium amount of horizontal scalability built into lemmy, and if the software doesn't fall over for algorithmic reasons (which is will at first), the current infrastructure architecture allows quite a lot of growth. There's plenty of time between now and a federation of million user instances to adopt a truly distributed storage backend if needed.

[–] aksdb@feddit.de 1 points 2 years ago (1 children)

Doesn't solve the availability issues, though. I know of no seriously hosted system that doesn't have at least two replicas in different availability zones. I don't expect any hobby instance to offer any kind of availability guarantee. But if we want to have one or two central instances that the typical reddit user can flock to, this would IMO be essential to have.

Also, in my experience it is FAR cheaper to have a few low to mid range systems for vertical scaling, than to throw a high end machine at it for vertical scaling. If you look the the pricing, the monthly costs for vertical scaling goes up exponentially once you want much more RAM and CPU cores (and storage, and so on).

Being able to scale horizontally solves both issues: hardware is cheaper and reliability is higher.

That lemmy is so damn efficient would then simply mean, that we can achieve excessively good results with low resources, where Reddit would already struggly and needs to put much more machines in place. That would be a nice "business" advantage.

[–] PriorProject@lemmy.world 1 points 2 years ago* (last edited 2 years ago) (4 children)

Doesn’t solve the availability issues, though. I know of no seriously hosted system that doesn’t have at least two replicas in different availability zones.

I'm not sure why you think the setup I've described can't have coverage in multiple availability zones. If the lemmy and lemmy-ui containers are stateless as I suspect, you can autoscale them. Pictrs is new to me, not sure there... but it appears to support object-storage which would likely make it stateless and the object-storage can replicate to multiple-az's. Postgres read-replicas can be placed in multiple az's as well. The only component that presents an issue is the Postgres write-leader, and failovers there can be done in minutes. Many many popular sites run with an infrastructure like this and achieve excellent uptimes.

I do get the power of horizontal scalability, I specialize in distributed databases. But they come at a cost in flexibility relative to something like Postgres... and we're very far from "needing" horizontally scaling database writes here. Everything else looks like it can be scaled horizontally if someone wants to take on the headache of doing so.

load more comments (4 replies)

[–] cecirdr@lemmy.ml 1 points 2 years ago* (last edited 2 years ago) (6 children)

I'm a noob. I created an account on beehaw and on lemmy.ml. That's because I see communities on one instance that I'm interested in and a different community on another instance. So if there's a technology community on both, how do I get to see all the technology posts without having to have two accounts?

This is really confusing for noobs like me. I'd just like to see one community to technology, one for Science, one for nintendo etc. I don't care it it's spread out amongst different servers to divvy up the load, but from the user side, it needs to be seamlessly integrated.

I'm still learning how all this works though. But I don't know how many folks that are more casual than me will be willing to figure it out. I hope they do though! It'll be worth it to leave reddit in the rearview mirror!

Edit: lawdy, I just figured it out. Local vs all on the communities list. It was right in front of my face. good grief!

load more comments (6 replies)

[–] interdimensionalmeme@lemmy.ml 1 points 2 years ago (2 children)

lemmy.ml should be a roundrobin dns that sends you to a random instance in the pool. Or else you will re-centralize lemmy and curmble under the IT bill.

[–] Neil@lemmy.ml 2 points 2 years ago

Except (as far as I'm aware) your account only exists on one instance. So, if I end up on beehaw.org due to the round-robin, my account on lemmy.ml will not authenticate to that instance. I would have to have a separate account per instance which is hundreds of accounts.

load more comments (1 replies)

[–] jarwinder@lemmy.ml 1 points 2 years ago (2 children)

is it possible to move an existing profile to a new server, like on Mastodon? or I need to create a new one and "start over"?

[–] Barbarian@lemmy.ml 2 points 2 years ago* (last edited 2 years ago) (2 children)

Right now, there is no import/export. It's a known useful feature, but the devs have no time to work on it (I've been following all the optimization work they've been doing on github, I don't know if they sleep). You'll have to start over atm, sorry.

load more comments (2 replies)

load more comments (1 replies)

[–] lightrush@lemmy.ca 1 points 2 years ago (1 children)

Point us to where the coin slot is. E.g. Patreon. We insert coin 🪙, you upgrade.

[–] nachtigall@feddit.de 1 points 2 years ago* (last edited 2 years ago) (2 children)

Here they list them:

https://join-lemmy.org/donate

EDIT: shoutout to the madlad who donated $1,000!

load more comments (2 replies)

[–] comfy@lemmy.ml 1 points 2 years ago* (last edited 2 years ago) (6 children)

@nutomic@lemmy.ml It might be a good idea to default the Communities page to All instead of Local, to help push users into discovering other instances and promote them.

load more comments (6 replies)

[–] ruud@lemmy.world 1 points 2 years ago

Lemmy.world is a new server, accepting signups. You're welcome there.

[–] hbar@lemmy.ml 1 points 2 years ago (1 children)

Hi, as one of the new people, is there a way to transfer to another instance or would I have to create a new account there?

[–] sexy_peach@feddit.de 1 points 2 years ago (1 children)

You have to create a new account. But that's easy ;)

[–] jarfil@lemmy.ml 1 points 2 years ago (2 children)

That's kind of wrong though, isn't it? What about stuff like GDPR data exports? Users should be able to export their data, then import it into another instance, effectively migrating instances.

[–] sexy_peach@feddit.de 1 points 2 years ago (1 children)

You are free to learn to program and write a user import routine for lemmy: https://github.com/LemmyNet/lemmy

[–] jarfil@lemmy.ml 1 points 2 years ago* (last edited 2 years ago) (1 children)

I know how to program, I also know how to wonder how many instances are running off the docker-compose with publicly exposed postgres... that would make import/export really easy, wouldn't it? 🙄

Anyway, would you say this isn't the right place to discuss this stuff?

[–] sexy_peach@feddit.de 1 points 2 years ago (1 children)

Why would this be not the right place?

[–] jarfil@lemmy.ml 1 points 2 years ago* (last edited 2 years ago)

You tell me, you sent me away.

I think data protection, retention, access, rectification and deletion laws are going to hit anyone hosting an instance. The EU is also in the process of introducing a "data migration" law, that is mostly targeted at "large" social media, but we'll see what ends up getting approved.

I'm not a compliance expert, but what I know about these laws makes me fear setting up an instance just to get hit by whatever fines.

[–] Packopus@lemmy.ml 1 points 2 years ago

You can on Mastodon, you just export your data, delete your account, create new account on another instance and upload your data and it's like what you said!

[–] TheYang@lemmy.ml 1 points 2 years ago* (last edited 2 years ago) (4 children)

Is scaling the server a largely financial issue, or not? @nutomic@lemmy.ml

could you reasonably confidently say that you could 10x the amount of users for something like 1000$/mo on liberapay?
If so, would you mind setting a "goalpost" for the community to help lift the financial burden?

load more comments (4 replies)

load more comments