Can Lemmy have distributed communities (lemmy.fmhy.ml)

submitted 1 year ago by Mastersord@lemmy.fmhy.ml to c/asklemmy@lemmy.ml

16 comments fedilink hide all child comments

I was thinking about this in regards to all the “defederation” posts.

Let’s say you spin up a server and over night it gets super popular and grows enormous. Now your yearly expenses shoot up and you’re forced to either look for a new host or shut down.

Now what if instead, you could get a few other people to spin up more small instances and distribute parts of your biggest communities to them, however the users don’t notice because The communities are looking across instances instead of within their home instance?

That’s the idea at least. This would allow for many things but most importantly, it would make things a bit more manageable. Thoughts?

you are viewing a single comment's thread
view the rest of the comments

[-] xavier666@lemm.ee 5 points 1 year ago* (last edited 1 year ago)

You are talking about horizontal scaling. It's something which is non-trivial and would need the core of the app to be written in a very different way. Think of the challenges

Your sub-instance and your friend's sub-instance which serves a common community can be geographically far apart
They need to be synced
Internal function calls which were once local now need to travel across the network
Depending upon the heterogeneity of the hardware, there can be a large jitter (bad for UX)

The admin of my instance lemm.ee has tried to make horizontal scaling by decoupling some lemmy functions but pure horizontal scaling is really hard (glusterfs/spark).

[-] ComradeKhoumrag@infosec.pub 0 points 1 year ago* (last edited 1 year ago)

Would it be possible to have the dns of the instance technically be a load balancer and just point to different instances of the same docker container, but keep the database as a single shared entity?

Edit: oh OP is talking about doing this on a sublemmy level instead of a site wide level. Would my idea work for the entire instance?

[-] xavier666@lemm.ee 1 points 1 year ago

dns of the instance technically be a load balancer and just point to different instances of the same docker container What you actually mean is the lemmy docker be loadbalanced. The DNS will only point to the loadbalancer.

This is still somewhat doable and easier than what OP is asking for but I'm unsure whether lemmy can handle the synchronization issues. My instance manager (lemm.ee) has kind of distributed his architecture by decoupling core components of lemmy. Example

DB on one server
Loadbalancer on another server
Main lemmy instance on one server

this post was submitted on 28 Jun 2023

30 points (96.9% liked)

Asklemmy

43992 readers

692 users here now

A loosely moderated place to ask open-ended questions

Search asklemmy 🔍

If your post meets the following criteria, it's welcome here!

Open-ended question
Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
Not ad nauseam inducing: please make sure it is a question that would be new to most members
An actual topic of discussion

Looking for support?

Looking for a community?

Lemmyverse: community search
sub.rehab: maps old subreddits to fediverse options, marks official as such
!lemmy411@lemmy.ca: a community for finding communities

~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~

founded 5 years ago

MODERATORS