this post was submitted on 19 May 2025
56 points (100.0% liked)

Fedia Discussions

2 readers
1 users here now

founded 2 years ago
MODERATORS
 

Hi all. Fedia.io has for a long time been subject to ddos attacks, including many that are "accidental", caused by myriad scrapers constantly hammering the site. I gave up on trying to play whack-a-mole with blocking them based on IP address (they do not honor robots.txt and do not use a conspicuous user agent string) since I was inadvertently blocking some legitimate users. So, I've restricted access to the content of fedia.io to only those that are logged in. That will mean we don't show up in search engines and whatnot, which for some will considered a good thing and will likely cause others to leave.

There is a remaining problem related to the login form. Calls to the login page are breathtakingly expensive, computationally speaking, and so I also have a script that monitors unusual numbers of calls to that form and blocks at the firewall any offenders. I strongly suspect I'm catching some legitimate users with this too, and so I continue to try to tune it, but it's maddening, y'all.

These issues have been causing performance problems for everyone (despite the fedia.io app running on a dedicated 96 core, 256GB server with nvme disks), and became unavailable for certain people that accidentally tripped various thresholds. I'm hoping most of this is resolved now.

Thanks for the patience.

top 24 comments
sorted by: hot top controversial new old
[–] kbal@fedia.io 19 points 1 month ago (2 children)

a dedicated 96 core, 256GB server with nvme disks

The hunger for CPU this software has is amazing, as is your generosity in running it for us all. I wonder what's wrong with the login form. It just looks like simple html from this side.

[–] melroy@kbin.melroy.org 11 points 1 month ago (1 children)

Yes it should be as simple as twig html templating. And some basic db queries like checking if you are logged in already or not.

So we try to optimize where possible now. And find the root cause of the unwanted high resource usage. Reducing the amount of db queries as well as if we can get rid of csrf sessions and migrate to stateless csrf.

Then again, if your server is indeed getting ddosed, like 100k requests per second, almost any application, server or database will be unable to cope with that amount of load.

[–] jerry@fedia.io 10 points 4 weeks ago (1 children)

It’s an application level ddos. Blocking anonymous access helped a bunch, but I am still getting about 5-10 login requests per second from hundreds of different IPs

[–] melroy@kbin.melroy.org 5 points 4 weeks ago (1 children)

I heard the improvements on the main branch already helped you.

[–] jerry@fedia.io 5 points 4 weeks ago

You and the mbin team continues to amaze me. Thank you so much!

[–] jerry@fedia.io 10 points 1 month ago

We think it’s a csrf prevention measure in the php symphony library that creates a lot of database calls.

[–] xep@fedia.io 16 points 1 month ago

Thanks for doing this, I really appreciate having this instance to call home.

[–] Nougat@fedia.io 13 points 1 month ago (1 children)
[–] jerry@fedia.io 10 points 1 month ago

Thanks. Just trying to give people some alternatives

[–] atro_city@fedia.io 13 points 4 weeks ago (2 children)

This instance is provided to you for free without ads. If you would like to contribute to the costs of running the instance, we have the following donation options: Patreon: https://www.patreon.com/infosecexchange

Ko-Fi: https://ko-fi.com/infosecexchange

Liberapay: https://liberapay.com/Infosec.exchange/

You can also support with a one-time donation using PayPal to "jerry@infosec.exchange".

https://fedia.io/about

[–] melroy@kbin.melroy.org 10 points 4 weeks ago (1 children)

And mbin is also provided open-source and free. Don't forget the contributors working on the software.

For me that would be: https://melroy.org/donate.html

[–] jerry@fedia.io 10 points 4 weeks ago

I will add that to the donation page

[–] jerry@fedia.io 6 points 4 weeks ago
[–] Australis13@fedia.io 8 points 1 month ago

Thankyou for fighting the good fight!

[–] chookity@fedia.io 4 points 1 month ago (1 children)
[–] jerry@fedia.io 7 points 1 month ago (1 children)

Not really. We have to accept incoming connections from thousands of other fediverse instances that would be blocked by that.

[–] melroy@kbin.melroy.org 4 points 4 weeks ago

What about anubis on the normal html pages but not the apis and fediverse paths? You could so check for the http accept header.

[–] ciferecaNinjo@fedia.io 2 points 3 weeks ago (1 children)

I’ve always appreciated your competence and diligence in setting a good example of responsible hosting without resorting to shitty technofeudal fiefdoms like Cloudflare. Nice to see you are standing your ground and not selling the users out (unlike lemmy world and many other boot licking hosts).

I must say there is a notable side-effect to this. Since mbin does not have a cross-posting feature, I have been cross-posting by creating a link post to my original post from other relevant magazines. Now all of those links are unreachable to outsiders. To outsiders, I polluted their magazine with access-restricted links.

I can think of only two workarounds:

  1. Make the original post on a publicly open forum, then link to that from other forums. This means the original post can never be on Fedia (which has the side-effect of reducing fedia publicity); and/or
  2. Copy/paste the payload of the original msg into the cross-post.

Fix 1 is impossible for existing past threads. Fix 2 is tedious and it’s a maintenance burden esp. if a post needs edits or updates. Fix 2 is also problematic because if I withold the original link, users cannot find other discussions that are scattered; but if I supply the original link, then non-Fedia readers cannot reach the OP anyway.

That will mean we don't show up in search engines and whatnot, which for some will considered a good thing and will likely cause others to leave.

Worth mentioning that paywall sites handle this by giving crawlers special treatment. I’m not necessarily suggesting that though.

There is a remaining problem related to the login form. Calls to the login page are breathtakingly expensive,

The login form loads must be through the roof because whenever a non-fedia user follows a cross-post into fedia, they are redirected to a login form which did not happen before.

There would be some relief if Mbin would implement a cross-post feature that automatically copies the OP text into the body, which would cut down on the number of visits. At the same time, I’ve always considered that a sloppy approach because edits are not sychronised. So in principle the threadiverse probably needs a smarter API specifically for cross-posts.

The use of 3rd-party clients would obviously give relief on the login form loading. But I have not found any decent 3rd-party clients for Lemmy or [km]bin - (perhaps because I’m fussy… I could really use a text UI in linux that stores content locally).

[–] jerry@fedia.io 4 points 3 weeks ago (1 children)

I understand. I have tried hard to make fedia.io work - it’s been far and away the most challenging app I’ve managed (note: the problems are all legacy kbin issues, the mbin team has been nothing but amazing). I am stuck in a difficult position - the site isn’t useful if I keep it locked down like it is now, and the site is super slow/requires constant attention if I make it open. I’ll have to assess my options and decide what the future for fedia is

[–] ciferecaNinjo@fedia.io 1 points 3 weeks ago

the site isn’t useful if I keep it locked down like it is now

I’d say it’s crippled but not useless, just as old-fashioned non-federated forums are still useful despite limitations. And as it is now we still have some of the fedi benefits.

bug 1

One bug comes to mind, which should perhaps be reported against kbin. Is the current locked down state something that is facilitated by the software, or did you hack it to redirect outsiders to login screens? If it’s the former, then the software is disservicing users who unwittingly post a link back to the access-restricted resource. If I cross-post by posting a link to fedia.io/yadayada, I should ideally get a warning to say “are you sure you want to post a link that is inaccessible to outsiders?”

bug 2 (more of an enhancement)

One work around is for a Fedia user to create a post, wait for a non-fedia response, then dig up the cached version on the non-fedia host and publicise that link in other places. That’s already possible with a bit of navigation effort. It would be useful if users could obtain a link farm of cached versions of any post or comment. Not just for the situation at hand but with small hosts coming and going coupled with censorship as well, users of mastodon, lemmy, and [km]bin all suffer from dataloss. A sophisticated client could use caching info to locally build/recover a complete thread, as well as track points of data loss.

Anyway, just brainstorming here.

[–] MHLoppy@fedia.io 1 points 3 weeks ago (2 children)
[–] jerry@fedia.io 2 points 3 weeks ago

Ohh - that is possible. I will check when I get back to my computer.

[–] jerry@fedia.io 2 points 3 weeks ago (1 children)

Apologies for the delay, but this is fixed now

[–] MHLoppy@fedia.io 2 points 3 weeks ago

Fedia.io celebrates We Love The King Day