lodion

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner 🙂

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

Its been a little while since I posted stuff :)

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments:
Not much has changed in quite a while. I still have a cron-job running to restart Lemmy every day due to memory leaks, hopefully this improves with future updates. Outside of that, CPU, memory and network usage are fine.
Object storage usage is growing steadily, but we're a long way from paying more than the monthly minimum Wasabi fee.

The upgrade to lemmy 0.19 has introduced some issues that are being investigated.. but we currently have no fixes for:

• thumbnails break after a time. Caused by memory exhaustion killing object storage processes.
• messages to/from lemmy instance not yet running 0.19 are not federating. I believe it requires bugfixes by the devs.

~~I've re-enabled 2 hourly lemmy restarts. Hopefully this will help with both issues, though it will result in a disruption to the site around every couple of hours.

When the hourly restarts are disabled I'll unpin this post. As any other issues are identified I'll post them here too. ~~

Update: I've disabled the 2 hourly restart after upgrading to 0.19.2... lets see how this goes...

Update2: no issues seen since the upgrade, looks to have resolved both the memory leak and the federation issues. Hooray :)

test

~~I'll be restarting AZ today for an update to lemmy 0.19.~~

Upgrade complete.

This is a major upgrade, so I expect there to be some issues. Strap in, enjoy the ride.

Expect:

• further restarts
• bugs
• slowdowns
• logouts
• 2FA being disabled
• possibly issues with images, upgrading pictrs to 0.4 at the same time

Hi all,

Looks like I'll be visiting Melbourne in winter 2024 for a wedding with my family. Any must-see tourist attractions you can suggest?
We'll definitely be hitting up Melbourne zoo, and maybe Werribee.

Thinking I'll have to hire a car.. though I'd prefer to stay central and use public transport and Uber around when required.

I'm kicking off a storage upgrade on the server, expect it to proceed in the next 15 minutes and be back online shortly after.

Here we are, another month down.. still kicking 😀

As usual I’m stating full dollar figures for simplicity, but they’re all rounded/approximate to the dollar. I’m not an accountant and its close enough for our purposes.

Income

$98 AUD thanks to 13 generous donors

Expenses

$45 OVH server fees
$10 ($~6 USD)Wasabi object storage
$39 ($25 USD) Domain registration extended to 6th August 2025
= $94

Balance

+$486 carried forward from July
+$98 income for August
-$94 expenses paid in August
= $490 current balance

Future

Baseline storage on the server is now at ~70% at the low point of the day, time permitting I'll upgrade shortly to ensure sufficient room for growth. Doubling the current storage will cost an additional ~$12 per month.

THANK YOU

to everyone that has contributed to the running costs of the site.

A very special thank you to @Nath@aussie.zone for everything he's doing here, both in public and behind the scenes. Without his help performing admin and moderator activities, this place would be a shambles. I've simply been too busy professionally and personally to pay as much attention as I need to. Thanks Nath, I (and I'm sure everyone else here) appreciate it 🤗

As always, if you have any questions please ask.

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Summary:

Not much to call out. The storage drop was due to purging a days worth of images, and clearing the entire object storage cache. When I have time, I'll upgrade the VPS to add storage.

Due to some disgusting behaviour, I've kicked off the process of deleting ALL images uploaded in the last day.

You will likely see broken images etc on aussie.zone for posts/comments during this period of time.

Apologies for the inconvenience, but this is pretty much the nightmare scenario for an instance. I'd rather nuke all images ever than to host such content.

CPU:

Memory:

Network:

The one spike here is from a DB backup being uploaded to object storage, prior to the upgrade to 0.18.4.

Storage:

Still ok here. You can see the daily minimum free space increase as I tweak the local cache for object storage.

Cloudflare caching:

Summary:

Nothing of note here. Once baseline storage hits ~70% and I decreasing the object storage retention period is no longer worthwhile, I'll upgrade the server for more storage.

Sorry this is a little late, I've been busy with real life this week.

I'm stating full dollar figures for simplicity, but they're all rounded/approximate to the dollar. I'm not an accountant and its close enough for our purposes.

Income

$267 AUD thanks to 25 generous donors, it is very much appreciated.

Expenses

$34 OVH server fees. Lower than normal, as OVH seem to switch services to calendar based rather than anniversary based from the second month. The July billing period covered 8 July - 31 July, August will be 1-31.
$40 Cloudflare. Due to huge volumes of egress traffic from Cloudflare in early July, I opted to pay for a month of premium in order to investigate. The cause was found and remediated. I've since dropped back to the free plan.

Balance

$193 surplus from July (Income - Expenses)
$293 carried forward from June = $486 current balance

Future

Wasabi's trial period expired on the 8th of July, so the first bill is due this month.

Domain registration, I'll be looking to extend by another year and keep it so that we always have at least 12 months paid up.

Server storage, I expect we'll need to upgrade for additional storage this month. With this upgrade I'll consider pre-paying/committing to the server for a longer period of time. This provides both a discount, and certainty to everyone that the money is put to good use.

THANK YOU

..again, to our generous donors. The offer of an @aussie.zone email address redirect is still open to any donors.

As always, if you have any questions please ask.