Lemmy Administration

737 readers
1 users here now

Anything about running your own Lemmy instance. Including how to install it, maintain and customise it.

Be sure to check out the docs: https://join-lemmy.org/docs/en/administration/administration.html

If you have any problems, describe them here and we will try to help you fixing them.

founded 5 years ago
MODERATORS
26
27
13
submitted 2 years ago* (last edited 2 years ago) by tko@tkohhh.social to c/lemmy_admin@lemmy.ml
 
 

Unfortunately, the official documentation on theming lemmy is severely lacking. I made some progress on getting it figured out today, so I wanted to share that with you all.

This is by no means meant to be an exhaustive guide, but my hope is that it will at least get you going. I'm sure that I will say things that are incorrect, so please correct me if you know better!

Background

Lemmy uses Bootstrap-compatible theming. As far as I can tell, this means that it uses a pre-defined set of CSS classes. This is important because if you provide a CSS file that doesn't have all of the correct classes defined, it will break the layout of your lemmy.

Your custom CSS needs to be saved in the bind mount for your lemmy-ui container. If you followed the install instructions on join-lemmy.org, the location will be /lemmy/volumes/lemmy-ui/extra_themes/.

Prerequisites

In order to generate the correct CSS, you need a couple of things:

  • your customized Bootstrap variables, saved in an scss file
  • the Bootstrap scss files
  • the SASS compiler

Let's go through each of these (last to first):

The SASS compiler

The SASS compiler needs to be installed on the machine you will use to generate your CSS files (it doesn't NEED to be the computer that lemmy is installed on, but it can be). Follow the install instructions relevant to you. I used the "Install Anywhere (Standalone)" instructions and installed SASS on the Ubuntu machine that is running my lemmy instance.

The Bootstrap scss files

These files need to be saved on the same machine as the SASS compiler. The Bootstrap download page has a button to download the source files ("Download source"). This will give you a zip folder, so unzip it. Within the unzipped files, the only directory you need to keep is /bootstrap-5.3.0/scss. Save that folder in a place that makes sense for you. I put it in my home directory, so the path looks like ~/bootstrap-5.3.0/scss. You'll need to reference this directory when you're creating your custom scss file.

Your customized Bootstrap variables, saved in an scss file

This is the fun part... you define your Bootstrap variables. I'm still a little unclear on which version of Bootstrap lemmy is using (and therefore which variables are valid), so I chose to start with one of lemmy's default themes as a starting point. I grabbed _variables.litely.scss and litely.scss from the lemmy-ui github repo as a starting point.

You'll notice that litely.scss is just importing variables.litely as well as the Bootstrap scss files. You'll need to change the path of the Bootstrap scss files to the path where you saved your copy of the files. However, leave bootstrap at the end of the file path, as this is actually referring to the bootstrap.scss file within the Bootstrap scss directory.

Generating the CSS file

Once you have all of the prerequisites satisfied, you can generate your CSS files using the SASS compiler. Go to the directory where your customized scss file(s) are saved, and run this command (you added the SASS install directory to your PATH, right??):

sass [inputfile.scss] [outputfile.css]

This will generate a CSS file. However, pay attention, as there might be errors. If so, fix the errors until you can run SASS without any errors.

Finally, drop the generated CSS file into your "extra_themes" directory. You'll now see your theme show up in the list of themes on your profile (it'll be the filename of your CSS file).


And that's it! I hope somebody finds this helpful. Please let me know if there's anything I can clarify!

28
29
 
 

UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.

Original post follows:


This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

  1. Remove custom emoji
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;
  1. Rotate your JWT secret (invalidates all current login sessions)
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();
  1. Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897

30
 
 

For anyone else running lemmy on kubernetes-

Here is an IngressRoute CRD you can use, to leverage your built-in traefik reverse proxy.

Normally-

(ingress / ingressroute) -> (service) -> (nginx proxy) -> (lemmy / lemmy ui)

With this-

(ingress / ingressroute) -> (service) -> (lemmy / lemmy ui)

A slight optimization to better take advantage of the built in kubernetes functionality. (since, it already has a nginx and/or traefik instance running).

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: lemmy
  namespace: lemmy
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`lemmyonline.com`) && (Headers(`Accept`, `application/activity+json`) || HeadersRegexp("Accept", "^application/.*") || Headers(`Accept`, `application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"`))
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`) && (PathPrefix(`/api`) || PathPrefix(`/pictrs`) || PathPrefix(`/feeds`) || PathPrefix(`/nodeinfo`) || PathPrefix(`/.well-known`))
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`) && Method(`POST`)
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`)
      services:
        - name: lemmy-ui
          port: http

Just- make sure to replace your host, with the proper instance name.

31
 
 

Lemmy.ml front page has been full of nginx errors, 500, 502, etc. And 404 errors coming from Lemmy.

Every new Lemmy install begins with no votes, comments, postings, users to test against. So the problems related to performance, scaling, error handling, stability under user load can not easily be matched given that we can not download the established content of communities.

Either the developers have an attitude that the logs are of low quality and not useful for identifying problems in the code and design, or the importance of getting these logs in front of the technical community and trying to identify the underlying patterns of faults is being given too low of a priority.

It's also important to make each log of failures identifiable to where in the code this specific timeout, crash, exception, resource limit is encountered. Users and operations personnel reporting generic messages that are non-unique only slow down server operators, programmers, database experts, etc.

There are also a number of problems testing federation given the nature of multiple servers involved and trying not to bring down servers in front of end-users. It's absolutely critical that failures for servers to federate data be taken seriously and attempts to enhance logging activities and triangulate causes of why peer instances have missing data be track down to protocol design issues, code failures, network failures, etc. Major Lemmy sites doing large amounts of data replication are an extremely valuable source of data about errors and performance. Please, for the love of god, share these logs and let us look for the underlying causes in hard to reproduce crashes and failures!

I really hope internal logging and details of the inner workings of the biggest Lemmy instances is shared more openly with more eyes on how to keep scaling the applications as the number of posts, messages, likes and votes continue to grow each and every day. Thank you.

Three recently created communities: !lemmyperformance@lemmy.ml -- !lemmyfederation@lemmy.ml -- !lemmycode@lemmy.ml

32
 
 

Hey, just went through a few different checklists, and discovered that Lemmy does not meet GDPR requirements for notifying users for how servers handle the data. I've brought up this request on github, and I hope to get it fixed soon, but in the meantime I've compiled a list of EU address blocks and intend to add them to my firewall. Just thought you all should know.

33
-1
submitted 2 years ago* (last edited 2 years ago) by Djokkum@rammy.site to c/lemmy_admin@lemmy.ml
 
 

I host a small Lemmy instance (just me at the moment), and on the Admin page i've noticed these rate limit settings. They don't seem to be documented and their meaning is a bit unclear to me. For example, what is the difference between 'Message Rate Limit' and 'Per Second'? And are these values per user or for the entire instance?

If anyone could explain these rate limit values (or point me to the relevant documentation) I would greatly appreciate it!

34
 
 

Hello, trying to use Ansible https://github.com/LemmyNet/lemmy-ansible to install Lemmy on a OVH VPS.

  • Configured DNS and ping it is ok.
  • I can reach the server with a sudo user on SSH

Installed Ansible on my local machine and follow the steps.

When i execute:

$ ansible-playbook -i inventory/hosts lemmy.yml

I have this error (I replaced for this help request real username and real ip address)

PLAY [all] ************************************************************************************************************************************************************************************************************

TASK [check lemmy_base_dir] *******************************************************************************************************************************************************************************************
skipping: [myuser@myip]

TASK [install python for Ansible] *************************************************************************************************************************************************************************************
fatal: [myuser@myip]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: myuser@myip: Permission denied (publickey,password).", "unreachable": true}

PLAY RECAP ************************************************************************************************************************************************************************************************************
myuser@myip      : ok=0    changed=0    unreachable=1    failed=0    skipped=1    rescued=0    ignored=0

What am i missing?

35
2
submitted 3 years ago* (last edited 3 years ago) by arbocenc@lemmy.ml to c/lemmy_admin@lemmy.ml
 
 

Hi,

I wonder what are suitable methods to protect a Lemmy instance against DDOS attacks.

For example, can we use Cloudflare? Or it could break the federation?

Any ideas/suggestions?

36
 
 

First of all. Your script is awesome. It's so fckn fast and absoluty simple and genius.

So i was wondering if some of you guys can help me install the script on my server too?

Greets Manuel

37
 
 

TL;DR: Do you think that a self-hosted instance of Lemmy might be usable enough for me to use as a discussion forum for my online classes?

Hi, I'm a HS teacher. Like many schools around the world, mine will be starting the academic year with a distance learning model.

I'm looking for a platform to host asynchronous discussions with my students this year.

I like the reddit/lemmy model of structured discussion and think it would be very useful for in a distance/asynchronous learning environment. I love the simple lemmy/reddit model of structured discussion for this. It's better than what I've seen in LMS's (flexible, easy to use, etc.)

Do you think that Lemmy might be usable enough for me to use as a discussion forum for my classes?

I would not be using Lemmy as the main LMS for my class, just as an occasional or ongoing discussion forum.

It would be a 'real world' use, so I care about usability and easy onboarding for non-technical users, and of course I don't want my server to break, BUT **the contents would never be mission-critical. ** Disruptions and even full collapse of the site would not be the end of the world. I would like the option of using Lemmy for certain kinds of assessment (did you contribute substantively? did you follow the norms of reasonable discussion that we talked about?), but I could totally cope if the worst were to happen... the point is the conversation.

Me: I'm a redditor with some slightly crufty experience as a linux sysadmin. I expect installation to be possible and documented, but I can troubleshoot, if you'd all be so kind to listen if I get stuck. Once the school year starts I won't have lots of time to invest in maintenance.

** Hosting: ** I'd host this on my own machine at home or (more likely) using a $5-$10 VPS.

Federation: I would not be planning to take advantage of the 'federated' aspect of Lemmy, at least at the beginning. An isolated site is better for this use case.

**Any comments or advice? Has anyone tried Lemmy for classes of students? **

Edit: If I have <150 users and pretty low usage overall, will 4gb of RAM be enough on the server? Would 2gb? ~~ I currently own something like the Value Server here 1 core / 2GB RAM / 40 GB storage? ~~ The Lemmy Install guide doesn't mention specs.

Edit 2: I currently have a ** Linode Nanode (1GB: 1 CPU, 25GB Storage, 1GB RAM).** This is certainly too little to run Lemmy, no?

38
 
 

looks like the build of diesel failed. I'm using a fresh install of Ubuntu 20.04. Which Ubuntu release are y'all using?

39