Post is 13m old; I see it from Lemm.ee but i see no other comments yet
Gives me hope :)
Here we are, coming back from lemmy.today!
Looks good from .world
Seems to be working pretty well so far.
Thanks devs, and thanks @mrmanager.
I haven't seen outbound federation issues (that I've noticed) since the 0.19.2 update. However....not an outbound federation problem, but I've been seeing a number of problems with not having comments from other instances show up in posts when viewed via lemmy.today.
In Jerboa, this manifests itself as a "there is no record of this comment" errors, with visible child comments beneath. In the Web UI, I don't see either the parent or child comment. I don't believe I saw errors like this prior to the 0.19.2 upgrade.
An example of one such comment (on a lemm.ee (0.19.2) community, !Wales@lemm.ee:
This appears fine in the lemm.ee Web UI for me:
https://lemm.ee/comment/8315979
There are presently two comments (a top-level comment from TWeaK@lemm.ee, and a child from Navarian@lemm.ee). Both comments are from users local to the instance. Both are over two hours old as of this writing.
In the lemmy.today Web UI, no comments are visible (though the post does have a comment count of two):
https://lemmy.today/post/5249421?scrollToComments=true
In Jerboa, when logged into lemmy.today, I can see the child comment, but not the parent comment. In place of the parent comment, there is italicized text reading "there is no record of this comment".
I'm running Jerboa 0.0.55, the latest on F-Droid as of this writing.
I'm assuming that the same problem is producing the output on both, and Jerboa isn't actually a factor in the problems, that the lemmy Web UI and Jerboa just act differently when a comment is "unreadable" (the Web UI hides the whole tree, Jerboa shows an error placeholder specifically for the comments it can't see and then tries to show the child comments), but I do not know this to be true for sure.
EDIT: A second example from another instance (feddit.de, also running 0.19.2):
Nine comments as of this writing, and are visible via the feddit.de Web UI:
https://feddit.de/post/7871940?scrollToComments=true
They're all visible on lemmy.world's Web UI:
https://lemmy.world/post/10751010?scrollToComments=true
But on lemmy.today's Web UI, no comments at all are visible:
https://lemmy.today/post/5235270?scrollToComments=true
In Jerboa, when talking to lemmy.today, I can see two comments, one from NoneOfUrBusiness, and one from DarkThoughts:
https://lemmy.world/comment/6715548
https://lemmy.world/comment/6716235
I get the "no record" error for several comments (hard to know which ones, since one can't see usernames, but due to the comment tree structure, one can infer that the top-level comment from ShroOmeric and the child from RidderSport are among those). Jerboa only actually attempts to display six comments for this post.
OK thanks for the detailed information! I really hate to see that Lemmy still has federation issues. It seems like one of those core things that would just work by now. But any info and help about remaining problems is very valuable for the developers. I will read through what you said here and see if I can create a github issue from it.
It seems like one of those core things that would just work by now
Well, IRC had netsplits decades in, so... shrugs
But yeah, I agree, it's a pain.
Thanks for all your work on this and the earlier problems, though.
Yeah true, it's always something. :) I used to be on irc too back in the day, but sadly not anymore. But the worse big tech becomes, the more we will want to go back to other tech that didn't bother us and felt good to use.
Just a wild thought, but are you still running that "restart lemmy periodically" cron job that you put in to deal with the earlier outbound federation problems? I am thinking that maybe the lemmy server software does not deal well with being shut down when it's trying to fetch a comment, and after it comes back up, never actually tries getting the comment a second time. That might explain why lemmy.today is seeing random comments not being pulled over, but it's not (at least immediately) apparent on other instances. I don't know what else would be special about lemmy.today.
I mean, it'd still be a bug in the lemmy server software, just something that maybe lemmy.today could be tickling more than other instances are if it's frequently restarting.
EDIT: Though if that's the cause, if it's also in earlier versions, that wouldn't explain why I hadn't noticed this earlier, when lemmy.today was running 0.19.1. I could have missed it in the Web UI, maybe, but it's pretty obvious when using Jerboa -- all those errors in place of comments -- and I think I would have seen it.
We dont have anything restarting the server automatically, but you are on to something.
There have been occurances of the lemmy software and postgres restarting now and then since the update (several times per day). Postgres memory usage has been enormous, and we turned off the extra web user interfaces for a while to see if its still happening. So far its actually looking a lot better and memory usage looks stable, so maybe our hunch is right....
There's some other issue with posts not propagating to lemmy.today. I'm not sure at what point this arose; I've only just noticed it.
I'm currently subscribed to !retrogaming@lemmy.world.
This is lemmy.today's view of that community:
https://lemmy.today/c/retrogaming@lemmy.world
The newest post to be visible is "Death Knights of Krynn [MS DOS] Intro on Creative Music System/Game Blaster", visible here:
https://lemmy.today/post/5957815
As of this writing, that post was five days old.
However, actually looking at the community on lemmy.world, there have been newer posts there. These have not propagated.
https://lemmy.world/c/retrogaming?dataType=Post&page=1&sort=New
As of this writing, there are four, the oldest of which is also five days old (so presumably, whatever caused the issue happened at least five days back, possibly exactly five days back). None of these have propagated to lemmy.today via federation.
https://lemmy.world/post/11464649
The Quest To Preserve Japanese Feature Phone Games
https://lemmy.world/post/11489415
Sly Cooper - What Happened to This Iconic Platformer?
https://lemmy.world/post/11526052
My Phantasy Star Online dilemma
https://lemmy.world/post/11531592
A more relaxing way to play text-based games?
The newer posts have propagated to some other instances that I have looked at, including lemmy.ml (0.19.3):
retrogaming@lemmy.world on lemmy.ml
feddit.de (0.19.2):
retrogaming@lemmy.world on feddit.de
feddit.uk (0.19.2):
retrogaming@lemmy.world on feddit.uk
Both lemmy.today and lemmy.world are reporting that they are federated with each other:
That is making me a little suspicious that whatever is broken is somehow specific to lemmy.today (and apparently an ongoing problem, not a one-off, since it's not just a single missing post).
I also have one request. Would it be possible to create a "lemmy.today community"? This could serve as a point for people to discuss or report issues on the lemmy.today instance (like this one). The closest thing we have right now is the "announcements" community, but post creation there is (understandably) locked for non-moderators. I'd do it myself, but I think that it'd probably be more appropriate were an instance admin the moderator. And while adding comments to this post is probably the best existing way to both notify you that there is an issue and make anyone else on the instance who might be experiencing the problem aware of it, I feel like most people aren't likely to go hunting into a recent update announcement to see if there are any issues on the instance or try to troubleshoot or help each other. I'd just message you directly with issues, but then other people can't see discussion or help or even be made aware that there is an issue.
On lemm.ee, there is such a community, with the name of !meta@lemm.ee.
I checked several other large instances, but wasn't able to find a similar-such community (I was looking to see if any convention had emerged as to what name to use for "discussion about the local instance"). I'm not sure that "meta" is the best term to use (I think it might be clearer to use the instance name or something, like !lemmytoday@lemmy.today, to make it clear that it's about the instance) but "meta" is short, and it has uptake from at least one instance.
Thanks.
Hi!
I looked and I can see the posts if I sort by New (default sort is Top Day I think):
https://lemmy.today/c/retrogaming@lemmy.world?dataType=Post&sort=New
That page has all the posts you mention above from the looks of it? Correct me if im wrong or if Im misunderstanding something.
And of course we can create a lemmytoday community, sounds like a great idea. A place to bring up issues about the instance. I agree with you about the name also. Meta doesnt sound good, and also its related to that Zuckerberg place... :)
Hi!
I looked and I can see the posts if I sort by New (default sort is Top Day I think)
Hmm. I'm sure that I checked sorted by New.
investigates further
Okay, this is more confusing. It looks like what happens is that when I am logged in to lemmy.today, I cannot see the newer posts via lemmy.today. When I am not logged in, I can. That means that it can't be a federation problem, that something else is going on.
I've got no idea what would cause that. I don't have my account set up to block anything at all. I don't think that there's some kind of remote per-user blacklist or even if there were, any idea why I'd be on it.
considers
If there is some sort of blacklist, I assume that I wouldn't be able to comment there.
checks
It looks like I can comment in the community there just fine, and that a comment made on lemmy.today promptly propagates to lemmy.world:
https://lemmy.world/comment/7332440
Hmm. Well, now I am definitely puzzled.
If you don't mind me asking, when you viewed the page and were successfully able to see the newer posts, were you logged in to lemmy.today? I'm half-inclined to go create a second account on lemmy.today to see whether it is also affected, but if your account is impacted as well, that's at least one more data point, and then I'm not just creating throwaway accounts.
My understanding is that lemmy very recently got per-user instance blocking added as a feature -- kbin has had this for some time -- and I am wondering if maybe something might not be working correctly related to that. I can't imagine what else would cause the user account involved to be an input.
Thank you for looking at this.
EDIT: Maybe if there is something in the stack that caches generated webpages and those aren't being invalidated, since user might be an input there, but then...no, that can't be it, because I also don't see the newer posts in Jerboa.
Hi,
I just checked and I see the same thing both logged in and logged out. Try clearing your cookies for lemmy.today completely, maybe its something weird going on with those? They changed authentication code in 0.19-something.
My apologies. This may just be a combination of an error on my part and Lemmy not working as I had thought it worked.
There are two retrogaming communities, one on lemmy.world and one on kbin.social. I had thought that I was subscribed to both, but I was only subscribed to the one on kbin.social. lemmy uses subscriptions to determine whether to cause posts to propagate.
What I still don't understand is why being logged is a factor. I definitely could see the posts when not logged in, so they made it to lemmy.today.
I just subscribed with my lemmy.today account, made a post with another account (on kbin.social), then unsubscribed and made another post with that second account. Both posts were visible to my account.
So I still don't understand why being logged in is a factor for the earlier posts, but at the time that I was seeing the problem, I wasn't subscribed, and lemmy doesn't guarantee that propagation happens in that scenario, so I'm not going to complain about that.
thinks
Maybe when you browse a remote community not logged in -- since it's not possible for a guest user to subscribe -- lemmy fetches the posts directly when the community is viewed, rather then having them pushed to it. Otherwise, it wouldn't be possible to do much as a guest user with remote communities. And if nobody on lemmy.today was subscribed, that would explain the lack of popagation. That'd produce that behavior, where a guest user can see posts, but a logged-in user can't unless they're subscribed.
I don't know why different logged-in accounts would see different sets of posts, but.
Well, I'm sorry for bothering you. Probably just user error, and me being too ready to flag something as a bug after seeing earlier federation breakage. Thanks for your time.
Its ok, Im happy if you find a bug in Lemmy so we can report it to the developers. There are many bugs on their github issues page, so I wouldnt be surprised if you encounter some.. :)
Was this an issue that could have been fixed by downgrading?
In theory yes, but it would have meant restoring a database backup and losing data that was created between the backup and the restoration.
We were considering it but it took a few days to realize it was a bug in Lemmy, and we always figured a hotfix would come quickly.
Unfortunately it was xmas and new years so it took a while for a fix to arrive. Next time we won't update the instance at such a bad time - lesson learned. :)
Mmm yeah that's tricky. Do all lemmy updates break compatibility with the database? (assuming that's why a backup restoration would have been required)
No its the first time they specifically said there was a db update in the 0.19.0 upgrade and it took quite some time to complete too. 30 minutes or so of downtime.
Anyway, feels like we are finally past the problems... Things seems to work normally again now.
Gives me hope :)
Announcements about system updates or other things related to this instance.