Lemmy.nz Support / Meta

Two weeks ago we moved to a new VPS, less powerful than the previous server but we have to pay now so it is what it is.

We also recently implemented Anubis, which should have cut the amount of processing power wasted on AI scraper bots.

I'm interested in knowing how people have found it.

Is it feeling sluggish or similar to the last server?

Images are now in object storage instead of on the hard drive, do you notice a difference in loading speed?

Is anyone still having trouble with Anubis and getting stuck on the success screen or have those issues all been resolved?

Any other feedback on the new setup?

Earlier in the series:

• We are losing our host and will need to move to a VPS.
• We moved to object storage for images over last weekend.
• On Friday we moved to the new host

TL;DR of this post: We now have Anubis on Lemmy.nz, which helps protect against scrapers.

Hi everyone! As some have noticed, the site has been slow these past few days, which was due to a massive amount of traffic presumed to be from AI scraping bots training LLMs. I posted this elsewhere but will put it here again for good measure (7 day history):

Yeah so that peak of the graph is over 2 million unique requests that day, and the earlier part of the week has about 100k a day. So a 20x increase in traffic.

You may have also seen some weird stuff over this weekend as I played with how to get Anubis inserted into our Lemmy setup, which wasn't too straightforward with our multitude of frontends and our load balanced Lemmy-UI. You might have seen errors, apps might not have been able to load due to being incorrectly blocked, or you might not have been able to load the site at all. Apologies for this, but I think I have it sorted now.

Running under the assumption that the scrapers are not directly targeting Lemmy but are just scraping everything on the net and we just happen to have a significant amount of content, Anubis has been added only to the front ends (websites you see accessing in a browser), and if you are logged in then the policies are set so Anubis should just pass you straight through without any checks.

The APIs and federation and anything else happening directly with the backend do not go through Anubis. This means apps should not be affected and loading images should not be affected.

I've already noticed a big difference in CPU usage. It went from periods of time averaging over 150% (as in, the CPU couldn't keep up, and you'd see Lemmy fail to load at these times), to the last hour peaking at 47% and mostly hovering around 15-25% CPU usage.

Click for image of last 24 hours CPU usage

Adding Anubis is a pretty big change that has a decent chance of breaking something so please let me know if you see anything weird!

I'm also happy to give more technical info if anyone wants to know about how it's implemented, the architecture of our Lemmy setup and how it fits in, or anything else. Just ask!

Earlier in the series:

• I have an update in this post. TL;DR we are losing our host and will need to move to a VPS.
• We moved to object storage for images over last weekend.

TL;DR of this post:

• We will move to a new VPS this weekend
• Lemmy.nz will be down (properly down) from sometime after 7:30PM Friday until the job is done. This should be completed within a few hours but I reserve the right to use Saturday as well 🙂
• Initial estimates are that the VPS will be around NZD $50/mo, on top of image storage costs (hard to guess - perhaps $10/mo)

I have now confirmed a plan with a VPS host Zappie, in their Auckland data centre. This is in fact the same place the existing server is colocated.

The specs of the new VPS:

CPU: 4 Cores Ram: 8GB Disk: 150GB Monthly Traffic: 1TB

We may need to adjust things as we go. Initially I will keep our current setup (including Cloudflare), but as we understand better what our needs are I am keen to move off Cloudflare as there was quite a lot of support for that move.

I have copied things across to test and it runs Lemmy well, similar to the current server, though that's without users, federation, and rogue AI scrapers but Zappie can bump specs pretty quick if needed.

Please let me know if you have any questions, I'm always happy to provide more details 🙂

In case you missed it, images have been down for the night while I did a migration from the hard drive storage to object storage.

The migration is now complete! Please let me know if you find anything odd with images.

In case you missed it, I have an update in this post. TL;DR we are losing our host and will need to move to a VPS.

It seems pretty clear that once we have to start paying for storage, it will be very expensive to keep everything on hard drive storage, so we will need to move to storing images on object storage.

Unless anyone has any objections, instead of doing everything in one go, I would like to do the migration to object storage for images in advance of the main move to a VPS in order to better assess what we will need for storage once that part has been offloaded. The plan is to use Backblaze B2 storage, because it's cheap, reliable, and already used for our backups.

This means we will need two patches of downtime, most likely this weekend and next as we need to be off the current host by the end of the month.

This post is to propose an outage from this Friday after 8pm, through to whenever it is finished on Saturday. I have been doing some testing, and it seems it will be a bit flaky if I try to do the migration too fast. My plan is to set it going the night before, run it overnight, and start Lemmy back up in the morning assuming the migration has finished by then. But there is every chance that it just stops while I'm sleeping and needs to be restarted in the morning, taking most of Saturday.

My current plan is to keep Lemmy itself running during this time, meaning there will be no images but federation will continue so we shouldn't need to catch up after it's done.

TL;DR

• Outage Friday after 8pm through to when it's done on Saturday
• Most likely Lemmy text content will remain available but no guarantees
• Moving image storage to Backblaze B2 - please tell me ASAP if there are objections (American company, but only real alternatives are Cloudflare or big tech and American anyway)
• A further outage will be required the following weekend to move the actual Lemmy service

As many of you will know, we have been borrowing server hosting, which kept costs very minimal. Many of you donated to the host (thanks!), but a while back they stopped taking donations as they wound down operations including transferring mastodon.nz to new people and gave me notice that we would eventually need to move to other hosting.

That was quite some time back, and now I've been notified that the time has finally come. At some point in this month we will swap over to a new host. It will also involve significantly increased costs to run the server (up from almost nothing), and most likely we will move to a VPS off of the current shared but dedicated hardware to more appropriate specs, and move to using object storage for images. All this will likely require some down time, which I will try to give notice of. I'm currently planning for this to be quite significant (perhaps done over the course of a whole weekend), since this would involve migrating all images to object storage which must be done with pict-rs shut down (others have reported 8+ hours for this), as well as transferring all other data to a new VPS, then a slow period of time as it catches up on federation. The second day would be a backup in case something went wrong.

When this move happens, I'll also start to do regular financial updates. In the past people have been interested in running costs, but they have been minimal out of pocket due to the server hosting being free to me. With the increased costs it makes sense to start regular updates about what it costs to run the instance, similar to what other instances do. There is a very wide range in what this could cost, and it's not clear right now what the lowest spec server is that we could get away with. But I am hoping we can keep this under $50 a month. People have asked previously about donations, a little while back I set up a page so you could donate directly to Lemmy.nz, here is the post about it (with the emphasis on not donating if it will materially impact you): https://lemmy.nz/post/21494039

Before moving forward, I'd like some input from Lemmy.nz users on some specific questions:

Currently we are hosted in NZ, how important is it to continue this? (with cost being the main barrier)

We will try to host in NZ if we can, but if cost becomes an issue, is it a big deal to host in Australia or elsewhere? We will want to try to stay close to NZ so NZ based users have low latency

Do people object to Cloudflare?

Currently we use Cloudflare, with benefits around loading faster for people in other countries (due to the CDN) and the ability to more easily handle AI scrapers.

Cloudflare puts us in the power of a large company and reduces our independence, but it also reduces bandwidth usage and have tools to more easily fight AI scrapers that cripple the server. Cloudflare offers captcha services that help this.

Remember, users on other instances that don't proxy images will also be loading images directly from our servers, which Cloudflare will reduce the latency of due to their CDN.

If people object to Cloudflare, I am willing to attempt to avoid using it, adding Anubus and managing IP blocklists, but it will be more work. It will also be a slightly higher cost, as we will need a higher traffic allocation since Cloudflare currently caches a lot of image traffic.

Pātai?

If you have any questions, let me know!

Earlier today I updated Lemmy.nz to Lemmy 0.19.15

Previously we werre running 0.19.13 so there are two versions updated.

Release notes for 0.19.14.

Release notes for 0.19.15.

Only minor changes. The main things seem to be:

• Recently some malicious users started to use an exploit where they would post rule violating content and then delete the account. This would prevent admins and mods from viewing the user profile to find other posts, and would also prevent federation of ban actions. The new release fixes these problems.
• Pages are now capped to 100, as in you can only go 100 pages of content clicking "Next" on the website, and the equivalent number of posts on apps. This is to reduce the risk of DDOS attacks.

The 0.19.15 release notes also mention Lemmy version 1.0 is coming soon and include a list of new features.

Hello Dave,

Sorry for bothering you again.

There is an issue with lemmy.world again

I only see one vote in this post https://lemmy.nz/post/30741653

Thanks for your time

Hello Dave,

It seems there is an issue with lemmy.world. Votes and comments are delayed, for an example this post: https://lemmy.nz/post/30741653.

I think this has been happening since yesterday, and I only realized it today.

Sorry about the down time, Lemmy.nz was just down for 30 mins.

I wish I had more info but I'm not sure what the issue was!

It seems our instance started getting hit hard about 5am, then we were at 100% CPU for the last 30 mins:

The number of hits has shot up too:

In case you're thinking well of course an NZ site has many more hits in daylight hours, here's the week:

I'm guessing AI scrapers?

Reposting this due to a grand total of zero votes from Lemmy.nz users...

It's been noticed that there is no Maps without NZ community on Lemmy. There's a short conversation here that suggests it's added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂 ^___^

Survey here: https://survey.lemmy.nz/index.php/892864

It's been noticed that there is no Maps without NZ community on Lemmy. There's a short conversation here that suggests it's added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂

We just had about 2 hours of downtime, sorry about that. It was an issue at the host end.

We borrow hosting, and the host had to make a trip into the data centre to get things going again.

With a length of downtime like this there will be a period of time while federation catches up, so expect some amount of delay with getting content from other servers.

Apologies for the bit of downtime. I’ve just updated Lemmy.nz to 0.19.12 as well as updating pict-rs which handles images.

We have upgraded two Lemmy versions (previously we were on 0.19.10, now we are on 0.19.12).

The first is 0.19.11. The release notes are here.

For 0.19.12, the release notes are here.

Some changes:

• The Lemmy developers added a donation dialogue for donating to them.
• When registration applications are approved (or denied), you will now get an email (yay!)
• Various accessibility changes and a new theme originally used on the instance Rblind.com

There are also heaps of minor changes mentioned in the release notes linked above.

Please let me know if you find anything not working right!

Looking around lemmy-federate.com I see lemmy.nz showing up in the error and not allowed states for many of the federated communities.

For instance communities I moderate

• !interesting@hackertalks.com
• !metabolic_health@lemm.ee
• !media_reviews@dubvee.org

As you can see it crosses over multiple instances.

Apologies for the bit of downtime. I've just updated Lemmy.nz to 0.19.10 as well as aligning our nginx config with the latest config from the Lemmy repository. We had been running an original one from when the instance started, with some custom changes. Now we should benefit from changes over the last couple of years.

Details of the latest release are here: https://join-lemmy.org/news/2025-03-19_-_Lemmy_Release_v0.19.10_and_Developer_AMA

Some highlights:

• Youtube thumbnails should be fixed
• DMs are now removed when banning user with content removal (i.e. deletes scammer "Nicole the Fediverse chick" DMs)
• Resolving some language setting issues

I was hoping to solve a video streaming issue but the update to the nginx config doesn't seem to have helped so I need to play with it some more. So there may be more brief outages tonight but I will try things in my test instance first.

Hi everyone, I finally did it! I got us upgraded to the latest version.

The release notes are here.

It's a minor release but the main things seem to be a fix for languages for new users, a security fix, and various other fixes as listed in the link.

Apologies for interrupting any evening browsing, I saw my chance and took it.

Questions always welcome 🙂

It would be nice if everyone would use tags or brief quotations, so I can see who is replying to who.

Modern forums are confusing my old brain because I'm used to the old BB code and old style of forum navigation, but a lot of things have changed in Lemmy, and also with proprietary forums used by large companies.

Sometimes I forget how to use features. I'm no better than a boomer who needs constant reminders on how to hold a mouse.

I think some people either don't care or they simply forget that you can highlight text with the mouse and then click "reply" to make a quotation. Maybe they are using a phone and it's too difficult to do this.

I've been getting a connection timeout every now and then

Hi all, welcome to Lemmy 0.19.8!

We had about 10 mins of downtime starting from 1:17 this afternoon as the database migrations ran. Apollogies if this interrupted your lunchtime scrolling, this was needed because the Controversial scores needed to be re-calculated as detailed below.

We have skipped a few versions, we were on 0.19.5, and so we have .6, .7, and .8 updates. A brief summary follows, with links to the Lemmy announcements.

Lemmy 0.19.6 changes:

• This version introduces parallel sending, which should solve our issues with Lemmy.world and remove the need for the extra VPS in Finland. Unfortunately Lemmy.world needs to update for this to work, so it will be a while longer.
• Changed sameSite cookie from Strict to Lax - I think this should help the issue where you are logged out from the browser a lot (especially in Firefox)
• There's a change in how "Controversial" is calculated, to use exponents instead of multiplication, to match how Reddit does it. Unfortunately the Lemmy documentation page seems not to list Controversial ranking among the other explanations, but there is some discussion here.
• Removed the "Trending" box from the home page. I kinda liked how it worked for us to find our local communities but oh well.
• This was a reasonably big release but mostly backend stuff - see link above for more info.

Lemmy 0.19.7 and Lemmy 0.19.8 are small bugfix releases, some of which fix issues introduced by 0.19.6 so we didn't see them. I don't see anything of interest to an average user but if you're interested take a look at the announcement posts linked.

Let me know if you see any issues, or have any questions.

TL;DR: Lemmy upgrade some time early in the new year. I also recently enabled image proxying, let me know if you noticed.

I thought I'd make a quick post just to explain the plan for updating Lemmy.nz to the latest Lemmy version and another recent change to image proxying.

Lemmy upgrade

Some of you make have noticed that Lemmy 0.19.8 just came out, and yet we are still on Lemmy 0.19.5.

There have been various bugs in 0.19.6 and 0.19.7 that while not critical, there was also no reason to rush the upgrade and I decided to wait for 0.19.8 which had a fix for some thumbnails not working that was broken in I think 0.19.6.

My plan is to give a little time to see if others have issues, and then upgrade to 0.19.8. Most likely this will be early in the new year. I don't believe there should be any significant downtime so I will probably just do it when I am ready. Before I do, I'll do a test run and see how long it takes, then decide then.

As always, I'll post on the Lemmy.nz Matrix chat before I do it, and if I am having any issues.

Image proxying

You may have noticed I have enabled imaged proxying some 3 or 4 weeks back. This means all images are loaded from the Lemmy.nz server instead of whichever instance is hosting it. Personally I have found there to be significantly less instances of an image not loading. I have also noticed some images load a bit slower than before, which I think is if the image is not already cached then Lemmy.nz has to download it, then you have to download it from Lemmy.nz, so this tends to happen more on smaller communities.

However, overall I feel like the experience is improved with image proxying turned on (even if it has doubled CPU usage of the server).

As always, happy to answer any questions and hear your feedback 🙂

Xi wants to experiment with using xi/drag person-independent pronouns on this account. So far xi likes using the pronouns, but xi's only written 4 comments and already someone accused drag of talking about Chinese politics when xi mentioned dragself. Xi knows xi's pronouns are confusing, so xi would like to put xi's pronouns in xi's displayname to clear things up.

But the character limit for display names on lemmy.nz is only 22 characters, and xi's handle already takes up 16 characters. Xi would need 24 characters to put xi's pronouns in xi's display name. Xi knows some instances also have a separate field for pronouns, and the pronouns get displayed next to the name without taking up displayname characters. Xi would like a place, or at least enough room, for xi's pronouns, please.