tal

Yeah, I don't want the whole world of social media to wind up in a "get an account everywhere to view anything" situation.

I think that it's a reasonable stopgap fix from an admin standpoint, because there aren't a whole lot of great levers for admins to pull to try to deal with bots beating the hell out of a server. And one can't just have one's server become unusable.

But it should really be something that is addressed on the dev side. Like, we need a real, long-term solution for the Web.

Also, while improving server performance and failure modes under load would help, the "badly-written, aggressive scraper bots that ignore robots.txt and are given loads of network resources clobbering servers" is something that affects many, many different Web servers out there. This isn't a Lemmy problem, nor even just social media problem. It's a Web problem.

I think what really needs to happen is some form of mechanism that makes it hard for purely-automated systems to hammer a server, something that segregates "bad bots" and humans. Maybe a CAPTCHA or other proof-of-resource thing or something like that.

It's going to also create server admin problems. Think of things like Web search engines indexing sites

server admins normally want their servers to be indexed by "good" bots

so it's going to create additional headache in that even after successfully creating a system that is able to segregate bots and humans, admins have to be able to identify "good" bots

maybe "good" IP ranges or have "good" bots use public keys or something

and then whitelist those. Maybe have some mechanism to distribute "good bot" whitelists and an Apache mod that downloads them occasionally, lets an admin subscribe to a whitelist or something.

I also kind of like the ability to occasionally wget -r sections of sites, and a solution of the above sort will break that without logging in in a browser and handing off browser cookies to wget, which is not ideal.

And there's some resource cost to humans

just lower

for proof-of-resource solutions, and human time cost for CAPTCHAs, all of which are undesirable.

considers

One unorthodox solution might be making it easier to snapshot a site. Like, okay. One of the things that irritates the hell out of me is that the bots are doing this to a number of websites that go out of their way to make it really easy to distribute the data that they're after.

Like, GitHub

which has lots of open-source code, which is useful to train coding AI models

ran into load problems with bots scraping it. That was induced by the same data being massively downloaded over via inefficient Web views. Bots would follow every single link, many of which displayed the same data formatted slightly differently. But...you can already get all of the code far more efficiently, which would be better for GitHub and better for bot operators, by just using git clone to pull the code (and all of its history!). Bot operators could just say "oh, this is GitHub" and just git clone everything and the load would be completely ignorable. I'm sure that people have mass-git-cloned GitHub lots of times before. The reason bot operators presumably aren't doing it is because it takes some amount of time and dev effort on their side compared to just "build generic Web spider and clobber everything".

Same thing for the Threadiverse. The Threadiverse will let you set up a Threadiverse instance and subscribe to everything, efficiently feed all the posts and comments you want to your instance, the moment they come in. In nice, machine-readable form, rather than in something intended for humans that you have to scrape and post-process. But...it takes more dev effort to set up something specific to the Threadiverse than to just treat it like another website.

If there were some sort of widely-adopted API for "request site snapshot" or "request snapshot of changes since time X", widely-enough that it were worth using, maybe bot operators would use that instead. If they don't have to write a "detect GitHub site and git clone" and "detect Lemmy and subscribe to posts and comments" system, but just have a single universal "dump changes" API, they might use that; less dev effort on their end.

considers

I guess the problem is that some website operators might treat the "snapshot API" itself as an opportunity to discriminate between bots and humans, and just return garbage or nothing as a snapshot. Some websites don't want to be scraped at all. If many websites returned incorrect data in response to such an API request, that'd kill the incentive of bot operators to use such an API.

It's probably not directly related to these changes if you're only doing it Saturday and it's only on old.lemmy.com, but if you're looking into load problems, might be indirectly relevant. In the past...I think two days, using the standard Lemmy Web UI, I've seen a high rate of the server returning a page just reading:

Error!

There was an error on the server. Try refreshing your browser. If that doesn't work, come back at a later time. If the problem persists, you can seek help in the Lemmy support community or Lemmy Matrix room.

Sometimes, but not always, the above error text is also followed by:

The server returned this error: Error. This may be useful for admins and developers to diagnose and fix the error

This has not, as far as I can tell, shown up when trying to view lists of posts (like, when I view all subscribed posts) but it does show up when trying to view the page for a post. It appears to affect all posts for me, not just specific ones.

I initially thought that its just that the post had been deleted, but I could view said posts fine on the original, remote instance.

One such example that I just hit, tried to load a number of times unsuccessfully:

https://lemmy.today/post/57605044

Which has, as a remote post:

https://nord.pub/c/games@lemmy.world/p/326761/is-routine-worth-playing

I reloaded five or six times, and eventually it came up locally on lemmy.today.

When it happens, it seems to affect all of the post pages that I try to load.

When I see this, the error page comes up pretty quickly, maybe in a second, so it's not some (long) timeout. Maybe hitting some limit on concurrent database queries or something like that?

tries loading page again

And now that post is back to showing an error. Also, I saw it without doing a cache-invalidating reload on Firefox (shift-reload rather than a normal reload), so whatever the server is trying to generate, and failing, it's not driven by the server trying to respond to something that my browser will have cached.

When the problem is happening, the same problem also shows up when I try to view my user profile page (so doesn't as best I can tell, affect lists of posts, either "all subscribed" or viewing posts in a community, but does affect individual post pages and does affect user profile pages).

There used to be a convention of having authors who would, at least occasionally, write deliberately-enraging opinion articles in magazines. In my experience, they were typically at the end of magazines. Maybe not outright wrong articles, but they'd make statements that many readers of the magazine would object to.

The idea, as I understand it, was to

and this was before social media, in the Internet-based sense of the word, developed a focus on this

drive engagement. People would write letters to the editor complaining and arguing against it, and then the magazine could print some of these in the next issue in the letters section.

John C. Dvorak was the most-prominent example of this that I recall

he put articles in a few computer magazines

and was notable in that I recall him publicly-admitting on one occasion that he did this, that it wasn't just that he happened to have controversial opinions.

Huh. Looking at his WP article, it looks like he just died last week.

Anyway, my point is that I could believe that publications might still have authors who do this sort of thing.

If you're using other mods at the same time, I'd try doing the binary search thing

turn them off, see if the problem goes away, if so, turn half of them off, see if the problem goes away, then of the problematic half, try turning half of that off, etc.

searches

It sounds like Bethesda says that something similar can happen in their games if you have a 5.1 configuration with nothing hooked up to some of the speakers.

I vaguely recall some bug on very old versions of Proton, something with the sound subsystem, that caused speech to not show up in some Bethesda game, though I think it was a problem that showed up after a while, not always not present from the start of the game. Trying to deal with that was what initially got me to use Proton GE. I doubt that it's what's going on here, but you could try switching Proton versions.

I don't have Fallout 4 installed at the moment, but Skyrim has separate volume sliders in the settings for voice and music, so I assume Fallout 4 does too. You might check that.

I personally would much rather have a PC, given a choice between having one or the other. When I have had both in the past, I have always wound up with a console that's gathering dust.

However, that depends on what you're doing with it and what you want. Consoles have their own strong points:

• Less setup/troubleshooting/maintenance.

• Because the platform is locked down, harder for other players to cheat in multiplayer competitive online games. The PC is open.

• Everyone has more-or-less the same hardware, which is typically desirable for 3D multiplayer competitive video games. No "get a more expensive and powerful video card to get a slight edge".

• There is basically one standard controller, so games in the game library are more likely to support the features of your controller (assuming, of course, that you're using a controller on the PC).

• Some platform-exclusive games (and obviously that cuts both ways, and the PC library is much larger, but if you very specially want some of the PS5 exclusives, you need a PS5 to do it).

IIRC from an article about low levels on the Rhine or somewhere a few years back, one issue that creates is that it reduces the theoughput possible for cargo shipping on the river.

searches

Sounds like it's an issue on the Rhine this year too.

https://www.reuters.com/business/environment/shipping-problems-continue-rhine-water-level-falls-again-2026-07-27/

Shipping problems continue as Rhine water level falls again

Shallow water after this summer’s heatwave and scarce rainfall means cargo vessels are only able to sail partly loaded on the river, increasing costs for cargo owners. It ​also means loads must be spread among several vessels sailing part ​loaded, also increasing costs and creating an unwelcome extra expense for German ⁠industry currently showing signs of recovery.

Vessels are ​often able to sail only 15% to 20% full or less, but sailings are continuing, ‌a ⁠commodity trader said.

They can't make competitive dedicated memory in a near-term timeframe

the kind of timeframe that would matter, if they are to get something in during the shortage

and Cook has explicitly said that Apple will not try.

However, they do make their own CPUs, and they could maybe make a large-onboard-cache version, kind of akin to what AMD has been doing. That'd make the CPU more amenable to use of slower memory for most applications, which might open up more memory options.

I don't know about a whole farm, but I suppose that the plant that I'd most personally be likely to grow would be some sort of hot peppers.

https://en.wikipedia.org/wiki/List_of_Capsicum_cultivars

Not a lot of work, and nice fresh in small quantities

so having some growing there is nice. Plus, it's something where a grocery store is only gonna have a limited selection of options relative to what's out there.

Whether that's a good choice is gonna depend on climate where one lives.

I also kind of like pineapple guavas. Those are hard to find in stores (unavailable in many places) as they don't ship well.

https://en.wikipedia.org/wiki/Feijoa

If I didn't personally have to take care of them, probably a variety of fruit trees, grapes, berries, maybe squash. Dad planted stuff around the back yard when I was a kid planned so that there was always something edible out there year-round. Grafted stuff on there too, so you had plumcots on one of the apricot trees, a peach tree that was half yellow and half white, a bunch of types of apples and plums (IIRC seven types on one tree) and so forth. It was neat from a consuming standpoint, but I wouldn't personally want to deal with caring for each. If you enjoy the hobby, though...shrugs More fun than just having a ton of one thing. Not optimal from a cash crop standpoint, though.

EDIT: Oh, strawberries. Not that they're hard to find, but store-bought strawberries are really watery, as they're sold by weight or something like that. If you've ever had strawberries that aren't as heavily watered, like, wild strawberries, it's a neat experience. Much stronger flavor.

We can desalinate all the water we want in California

we already have a bunch of desalination facilities, and can build those out as far as need be

but desalinated water isn't as cheap as already-fresh water. That's mostly an annoyance for, say, residential use. But it's harder to competitively do agriculture on desalinated water. Israel has done some limited agriculture on desalinated water, but farmers are not generally going to be okay with that as an option. Most agriculture uses a lot of water.

searches

Someone ran the numbers to see roughly what it would cost if we did only desalinated water for everyone in the US.

https://hannahritchie.substack.com/p/how-much-energy-does-desalinisation

Producing enough drinking water for someone — assuming 3 litres per day — costs just $2.30 for the entire year. That’s less than the cost of a single bottle of water in many countries.

We actually use way more water than we do for drinking

we're pretty prolific with water to do things like lawns.

If the average person in the United States uses 310 litres of water per day for domestic uses, it would cost them $0.42 per day. Or $154 per year.

We'd probably want to conserve more nationally in such a scenario (Californians use less than the national average). But even with no conservation, it's very much doable.

It's a cost that we'd rather not pay if we don't have to, as a society, but we certainly are able to do so for residential needs.

But if there isn't enough freshwater to do agriculture, then who gets the freshwater determines which farmers go out of business (or at least have to shift to something else) and which get to keep operating.

That's a good sign that they're going to sell enough product that it won't be a flop. I was pretty pessimistic on their chances of being able to introduce new hardware successfully during the RAM shortage, but perhaps I will be wrong!

That's pre-atomic-weapons-testing steel right there.

https://en.wiktionary.org/wiki/close_enough_for_government_work

Adjective

close enough for government work (not comparable)

1. (US, idiomatic, humorous, derogatory) Adequate; acceptable as government work, imperfect but not worth the time nor effort of improving.

Synonyms: good enough for government work