785
There is a drop in monthly active Lemmy users (from 65k to 57k)
(lemmy.fediverse.observer)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 22 Aug 2023
785 points (95.8% liked)
Fediverse
17652 readers
91 users here now
A community dedicated to fediverse news and discussion.
Fediverse is a portmanteau of "federation" and "universe".
Getting started on Fediverse;
- What is the fediverse?
- Fediverse Platforms
- How to run your own community
founded 4 years ago
MODERATORS
It's really odd how many people around here think the server crashes are perfectly normal and are glad to see newcomers driven away.
They are perfectly normal. Unlike giant corporations, the people who run Lemmy don't have the money to support a fleet of failover servers that take over when the main server goes offline. That's basically the only reason you don't see lots of downtime from major corporations: investment in redundancy, so when something breaks, a perfect copy takes over. Server crashes happen all the time for major corporations, you just never see them due to investment in redundancy.
That's the difference between a community and a company. One takes actual investment from the community as a whole, and the other ruthlessly exploits for profit.
That has nothing to do with the issue I'm talking about. Every server with the amount of data in them would fail. Doesn't matter if you had 100 servers on standby.
The Rust logic for database access and PostgreSQL logic in lemmy is unoptimized and there is a serious lack of Diesel programming skills. site_aggregates table had a mistake where 1500 rows were updated for every single new comment and post - and it only got noticed when lemmy.ca was crashing so hard they made a complete copy of the data and studied what was gong on.
Throwing hardware at it, as you describe, has been the other thing... massive numbers of CPU cores. What's needed is to learn what Reddit did before 2010 with PostgreSQL.... as Reddit also used PostgreSQL (and is open source).
Downtime because you avoid using Redis or Memcached caching at all costs in your project isn't common to see in major corporations. But Lemmy avoids caching any data from PostgreSQL at all costs. Been that way for several years. May 17, 2010: "Lesson 5: Memcache;"
As I said in my very first comment, server crashing as a way to scale is a very interesting approach.
EDIT: Freudian slip, "memecached" instead of Memcached