Hey folks!

I have deployed an unreleased Lemmy optimization on lemm.ee which significantly speeds up the query for loading posts.

So far, all seems good, but if you notice any issues with any of the post feeds, please let me know!

For more context:

Over the past few days, I have noticed a serious degradation for front page load times for some users. I have been trying different optimizations without much success, until I found an idea by @phiresky@lemmy.world to make a relatively small change to the database query for loading posts.

I implemented this idea and made a PR to Lemmy, you can check my PR here: https://github.com/LemmyNet/lemmy/pull/3653. Initial results seem amazing, my personal subscribed feed now loads extremely quickly, and I'm seeing from monitoring that load times should be improved for other lemm.ee users as well!

Hey folks!

I think I usually write too much, so I will try to keep it short and sweet this time.

Discord

I have created an official Discord server for lemm.ee! This is mostly intended as a back-up channel to share announcements with users - for example, if there is ever an incident and lemm.ee is offline, I can send updates about the situation in Discord. But feel free to join if you just want to chat with other lemm.ee users as well!

You can join the Discord at this link: https://discord.gg/XM9nZwUn9K

New admins

Two new admins have been added to the team! @Matt_Glan@lemm.ee and @Aris@lemm.ee have stepped up and volunteered to help me take care of the report queue. I think having good admins is super important to ensure a pleasant experience for all users, so I'm super happy that we have users who were willing to share this responsibility with me.

My hope is that adding a few more admins has helped ensure that I am less of a single point of failure for lemm.ee now. I am still considering maybe adding one or two more admins in the near future, but in terms of actual workload, I think the current team is already a pretty good size.

Anyway, that's all I have for you for now - please join the Discord if that's something you're interested in, and please welcome our new admins!

Edit: @prefix@lemm.ee has been added as an admin as well!

For now, I believe this is a good size for the team, we won't be adding any more admins in the near future. A big thanks to all who responded to the call for admins!

I think for a while leading up to the recent session stealing hack, there has been a massive amount of positivity from Lemmy users around all kinds of new Lemmy apps, frontends, and tools that have been popping up lately.

Positivity is great, but please be aware that basically all of these things work by asking for complete access to your account. When you enter your Lemmy password into any third party tool, they are not just getting access to your session (which is what was stolen from some users during the recent hack), they also get the ability to generate more sessions in the future without your knowledge. This means that even if an admin resets all sessions and kicks all users out, anybody with your password can of course still take over your account!

This isn't to say that any current Lemmy app developers are for sure out to get you, but at this point, it's quite clear that there are malicious folks out there. Creating a Lemmy app seems like a completely easy vector to attack users right now, considering how trusting everybody has been. So please be careful about what code you run on your devices, and who you trust with your credentials!

UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.

Original post follows:

This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

1. Remove custom emoji

DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;

2. Rotate your JWT secret (invalidates all current login sessions)

-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();

3. Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897

Hey folks!

So far, I have been the sole admin at lemm.ee. Most reports I receive don't really require any action from me - they are handled directly by moderators of the communities the reports originate from. Still, there is a chance that any reported post might contain content which would need to be purged from lemm.ee servers, so even reports that don't need any action should still be checked by an admin.

The volume of incoming issues has been quite manageable so far, but I have noticed a steady increase in daily reports lately. Additionally, there is nobody covering the report queue while I am sleeping or otherwise unable to access my computer.

I would like to ensure that there is a bit more redundancy in the admin team, so I am looking to potentially add one or two admins. Specifically, I am looking for somebody who would be willing to share the following responsibilities (copied from our administration policy):

Admins

• Ensure that there are no communities on lemm.ee which break lemm.ee rules
• Ban lemm.ee users who break our rules on other instances
• Ban users who consistently break rules across multiple communities
• Purge illegal content from lemm.ee

Note: I am not looking for help with system administration work at this time, this is strictly about administration within Lemmy itself.

Please be aware that being an admin is unfortunately quite a thankless job - if you're doing your job well, then most people won't even realize you're doing anything. OTOH, if you make mistakes, there will likely be many users calling you out in public. The main motivation for joining the admin team would need to be a desire to help build and maintain this instance as a great home for yourself and others.

If there is anybody who would be interested in helping out even despite the above disclaimer, please leave a comment with the following info:

• On a typical day, during what hours are you active on lemm.ee (with timezone info)
• Do you have any previous experience with moderation/administration
• Are you in agreement with the current state of the lemm.ee administration policy

Context

There have been a lot of posts and comments recently about Facebook entering the fediverse, and how different instances will handle it. Many people have asked me to commit to pre-emptively defederating from Threads before they even implement ActivityPub.

The lemm.ee federation policy states that it's not a goal for lemm.ee to curate content for our users, but we will certainly defederate any server which aims to systematically break our rules. I want to point out here that Facebook makes essentially all of its money from advertising, and lemm.ee has a no advertising rule - basically, Facebook has a built-in financial incentive to break our rules. ActivityPub has no protections against advertising, so it's likely we will end up having to eventually defederate from Threads just for this reason alone.

However, I would still like to get a feel for how many people in our instance are actually excited for potential federation with Threads. While personally I feel that any theoretical pros are by far outweighed by cons, I do want to use this opportunity to see how much of the community disagrees with me. I am not intending to run this instance as a democracy (sorry if anybody is disappointed by that), but I would still like to have a clear picture of user feedback for potentially major decisions such as this one. This is why I am asking every user who wants lemm.ee to federate with Facebook to please downvote this post.

Here are some reasons why I personally believe that Threads will have a negative effect on the fediverse

• As mentioned above, Facebook is completely driven by ad revenue. There is nothing stopping them from sending out ads as posts/comments with artificially inflated scores, which would ensure that their ads end up on the "all" page of federated servers.
• Threads already has more users than all Lemmy instances combined. Even if their algorithms don't apply to the rest of the fediverse directly, they can still completely dictate what the "all" page will look like for all instances by simply controlling what their own users see and vote on.
• Moderation does not seem to be a priority for Threads so far, meaning that they would create massive moderation workloads for smaller instances.
• In general, Facebook has shown countless times that they don't have their users best interests in mind. They view users as something to exploit for revenue. There are probably ways they are already thinking about hurting the fediverse that we can't even imagine yet.

By the way, we're not really in any rush today with our decision regarding federation

• Threads does not have ActivityPub support yet today
• Even if they add ActivityPub support, their UX is geared towards Mastodon-like usage - it seems unlikely that there would ever be proper interoperability between Threads and Lemmy
• We don't really know what to defederate from - it's completely possible that "threads.net" will not be their ActivityPub domain at all.

So go ahead and downvote if you feel defederation would be a mistake, and feel free to share your thoughts in the comments! It would be super helpful to me if folks who are in favor of federating with Threads could leave a comment explaining their reasoning.

Update:

By now, it's clear that there is a group of users who are in favor of federating with Threads. The breakdown is like this (based on downvotes):

• lemm.ee users: 136 in favor of federating with Threads
• Others: 288 in favor of federating with Threads

While it seems to be a minority, it's still quite a few users. There is no way to please all users in this situation - any decision I make will certainly inconvenience some of you, and I apologize for that.

A big thanks to everybody who has shared opinions and arguments in comments so far. I think there are several well written comments that have been unfairly downvoted, but I have personally read all comments and tried to respond to several as well. I will keep reading them as they come in.

The main facts I am working with right now are as follows:

• The majority of lemm.ee users are strongly opposed to immediately federating with Threads
• Facebook has a proven track record of exploiting users (and a built-in financial incentive to do so)
• We currently lack proper federation/moderation tools to allow us to properly handle rule breaking content from Facebook

Considering all of the above, I believe the initial approach for lemm.ee should be to defederate Threads, and then monitor the situation for a period of time to determine if federating with them in the future is a realistic option

In order to federate with them, the following conditions would need to be fulfilled:

• There needs to be actual interoperability between Threads and Lemmy
• Threads needs to prove that they are not flooding instances with rule-breaking content (mainly ads and bigotry for lemm.ee)
• There needs to be a mechanism to prevent feed manipulation by Threads algorithms (potentially this means discarding all incoming votes from Threads)

Note: this is an initial list, subject to change as we learn more about Threads.

Again, I realize this approach won't please everybody, but I really believe it's the best approach on a whole for now. Please feel free to keep adding comments and keep the discussion going if you think there is something I have not considered.

I'm a huge fan of SNES-era RPGs (and modern games which build on that style), so this looks like something I would really enjoy. There's even a big sale on Steam, I'm definitely going to try it out.

Thanks for helping me discover something new! 😃

👋 to all the newcomers, let me know if you need any help getting settled in!

Hey lemmings!

I wanted to share a quick update about our recent performance issues and how I have addressed them.

The last 24h have been a bit rough for lemm.ee.

Last night, I spent some time debugging federation issues with lemmy.world. We managed to significantly improve the situation - lemmy.world content is now reaching lemm.ee with a very high success rate - but this has had the effect of increasing incoming federation traffic on our servers significantly.

Additionally, we have been seeing steadily increasing normal user traffic over the past week, which is awesome from a community standpoint, but of course means that our servers have to do more work to keep up with all the new people.

To top things off, today there appeared a badly configured instance in the network, which was effectively launching a DoS attack against lemm.ee for several hours. Most likely it was unintentional, but unfortunately the end result was a sudden increase in our server load.

All these factors combined resulted in a really bad experience for most lemm.ee users today. Page load times have consistently been spiking into as much as 10 seconds or more for the whole day:

In fact, a lot of page loads just timed out with errors.

Fortunately, it seems I have managed to clear up the problems!

I have put a bunch of mitigations in place, and after monitoring the situation for the past hour, it seems that our performance issues have been resolved for now. So hopefully, you can enjoy browsing lemm.ee again without it feeling like torture!

Here are specific steps I took:

• I have doubled the hardware resources for our backend servers and database.
• I purchased a Cloudflare pro subscription for lemm.ee for 1 year. This took out a considerable chunk of my budget for lemm.ee, but in return it will allow me to analyze and optimize our cache usage to a far greater extent. I am already seeing vastly reduced load times for cacheable content (try opening https://lemm.ee a few times in a row as a logged out user - it should be blazing fast now!)
• I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.

Of course, all of the above is costly. Luckily, lemm.ee users have been very generous with donations in the month of June, and in fact a significant amount of donors have opted for monthly recurring contributions. This all gives me the confidence to increase our spending for now, and I am currently expecting to NOT increase my personal planned contribution of 150€/month, as the increased costs so far are entirely being covered by donations!

Let me take this opportunity to thank the sponsors who made the upgrades possible! All lemm.ee users are now enjoying better performance thanks to you, I could not have done it without you awesome people.

On a final note, I just want to say that I hope a lot of these issues can be solved by optimizations in Lemmy software itself in the future. I have been personally contributing several optimizations to the Lemmy codebase, and I know many others are focused on optimizations as well. Just throwing extra resources at the problem will probably not be a sustainable solution for very long 😅. But I am optimistic that we are moving in the right direction with the software changes, and we'll be enjoying reduced resource needs before long.

That's all I wanted to share today, I wish you all a great weekend!

Update: The maintenance has been completed!

Welcome to 0.18.1! Hopefully you'll notice some much needed UI tweaks as well as several performance optimizations.

I am still going through things and making sure everything looks good, but so far, I have not detected any major issues (fingers crossed). If you do have any problems, please comment below!

Original annoucenemnt below:

0.18.1 is coming!

As I mentioned in a previous post, this upgrade will require brief downtime. I will try to keep it as short as possible, but I can't guarantee an exact window, so if you really can't wait for lemm.ee to be back online, then the best advice I can give is to check back every 5 minutes or so.

As with any update, please be prepared for unexpected side-effects. If we find any major issues, then we can always roll back to good old 0.17.4. If we just find some minor issues, then most likely the overall experience will still be better on 0.18.1, so in that case we stay on 0.18.1 and try to get any potential issues fixed in 0.18.2. In either case, if you see anything weird after the upgrade, please report in the comments!

Tee, mis minu koduni viib, on iga suvi paksult rattureid täis. See tee on piisavalt kitsas, et kui 2 autot + rattur kolmekesi kõrvuti on, siis ratturil on suht suur oht peegliga pihta saada ja kraavi sõita.

Mul tuleb vähemalt korra nädalas ette olukord, kus hakkan mõnele ratturile tagant järgi jõudma, samal ajal näen, et vastassuunast tuleb auto. On aru saada, et see auto kulgeb rahulikult enne ratturist ja minust mööda, kui ma ise ratturini jõuan. Valmistun juba suunatuld sisse panema, et vastassuuna vabanedes kohe ratturist möödasõitu teha... ja siis saan aru, et vastutulev auto on hakkanud pidurit vajutama. Pidurdab hoo täpselt nii palju maha, et me jõuame enam-vähem samal hetkel ratturini. Mis mul muud üle jääb, pidurdan siis ka hoo maha, ja ootan ratturi taga, et tee möödasõiduks vabaneks. Tulemus on see, et kõik osapooled (peale ratturi) kaotavad aega ja kulutavad pisut rohkem kütust jne.

Eriti hull on siis, kui selle vastutuleva auto taga on pikem kolonn, kes siis keegi enne kiirendada ei saa, kui esimene jälle gaasipedaali üles on leidnud.

Mis värk sellega on? Kui ise vastassuunas ratturit näen, siis ei tule küll pähe, et peaks hoo maha pidurdama 😃

/rant over