I know there's been a few posts about this before, but it's been a month since the last one and it's still ongoing. It doesn't seem that any of the LW admins responded to Zag's post on their help community, and the last response from lodion/Nath I'm aware of was from 4 months ago when there were outright federation failures as opposed to just lengthy delays.
@Nothing4You@programming.dev posted a comment on the post from last month about the delays stating that it's an issue on our end as our server isn't keeping up. I'm not sure whether this is the case or not, and I'm not sure how to interpret the Grafana dashboard they linked to, but as it's a new reply on an old post, I wanted to note it.
Current federation delays seem to be around 7 days. It doesn't seem to be affecting posts themselves on Lemmy.world communities, but does affect all replies to them (even from users on other instances), and all upvotes on the posts. [Edit: on further investigation, this isn't the case. The current delays are at least 13 days, and this does actually affect posts too]
I don't want to sound too pushy, since the LW admins and Lodion/Nath are all volunteers, but I was hoping we might be able to get an update on what the cause is, and if it's an issue in Lemmy itself, if anybody's opened an issue on GitHub and the developers are aware.
(NB: I don't interact that much with LW, so all of my testing has been on the Boost for Lemmy community.)
this isn't exactly an issue in your end, unless you consider hosting the server in Australia as your issue. the problem is the latency across the world and lemmy not sending multiple activities simultaneously. there is nothing LW can do about this. as unfortunate as it is, the "best" solution at the time would be moving the server to Europe.
there are still some options besides moving the server entirely though. if you can get the activities to lemmy without as many delays am experience similar to being hosted in Europe can be achieved.
I believe the NZ instance basically set up a proxy right next to the LW server, which catches the ActivityPub messages and forwards them in batches to another proxy located in NZ, which translates back into a Lemmy-legal synchronous format. Rather a convoluted process, but it seems to work:
yes, that's about the second best option for the time being.
it's currently used by reddthat.com and lemmy.nz.
disclaimer: i wrote that software.
Oh, ha! Sorry for the lemmysplaining!
Is that software available for others to use? I've been busy lately, but Nath messaged me about this... I was traveling and forgot about it by the time I returned.
I'm open to the idea of setting up an EU VPS to proxy LW traffic temporarily. But the best fix is of course for Lemmy to operate in a way that doesn't cause this.
it's open source: https://github.com/Nothing4You/activitypub-federation-queue-batcher
I strongly recommend fully understanding how it works, which failure scenarios there are and how to recover from them before deploying it in production though. not all of this is currently documented, a lot of it has just been in matrix discussions.
I also have a script to prefetch posts and comments from remote communities before they'd get through via federation, which would make them appear without votes at least, and slightly improve processing speed while they're coming in through regular federation. this also doesn't require any additional privileges or being in a position to intercept traffic. it is however also not enough to catch up and stay caught up.
this script is not open source currently. while it's fairly simple and straightforward, i just didn't bother cleaning it up for publishing, as it's currently still partially integrated in an unrelated tool.
I previously tried offering to deploy this on matrix but one of my attempts to open a conversation was rejected and the other one never got accepted.