554
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 01 Aug 2024
554 points (97.8% liked)
Technology
57944 readers
2909 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
Honestly, any platforms hosting user-generated content who use the legal argument that they only provide hosting and aren't responsible for what their user post shouldn't also be able to sell the same data and claim owning any of it.
Otherwise, take away their legal immunity. Nazis or pedophiles post something awful? You get in front of the judge.
edit: typo
Exactly this. You can claim that their scraping is abusing your servers, but the moment you claim copyright for the content of the site, then you give up your Section 230 rights.
You'd also probably lose a whole lot more processing power trying to stop the crawlers vs just letting them have API access with some sort of limit to queries.
Eh, not really.
I block bot user agents to my Lemmy instance, and the overhead is pretty negligible for that (it's all handled in my web firewall/load balancer).
Granted, those are bots that correctly identify themselves via user agent and don't spoof a browser's.
It's also cheaper and easier to add another load balancer than size up or scale out my DB server to handle the bot traffic.
Systems admin here: You'll lose almost the same amount of processing time looking up limits and providing a "you're over your credits / rate limit" as you would by just providing the data.
Also, everyone will game the system with multiple accounts (cost be damned, the entities who want the data have cash to burn).
I don't think they actually block malicious bots, the change they've made is just to the robots.txt, they don't have to do anything.
Robots.txt does literally nothing. It's a piece of courtesy that's easily ignored if you don't care.
Yeah but it stops bing and a bunch of AI scrapers that want to act like they're following the rules
How do we know it stops bing? As far as anyone knows they could have instructed their programmers to alter the crawlers so that it ignores robots.txt when on Reddit - that should have taken them a whole 2 minutes.
Reddit blocking any search crawler via robots.txt is such a non thing that it shouldn't even be reported.
Ok but they do respect it, we know that https://searchengineland.com/microsoft-confirms-reddit-blocked-bing-search-444385
They even have a page telling you how to use it https://blogs.bing.com/webmaster/June-2008/Robots-Exclusion-Protocol-joining-together-to-pro/
Can't sell something you don't own.
So if they're selling the parts people want, they need to own the parts no one wants.
Well, you can give money to Reddit for a piece of paper, but unless Reddit is claiming copyright to the content posted there, then they can't sue anyone for not paying. It would be very interesting to see the text of these "licensing agreements".
They're not claiming copyright. They have a perpetual, non-revokable license to the content, granted by the people who use their site when they post the content.
Good point!