This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
% grep lemmy Meta.txt
lemmy.ca
lemmynsfw.com
lemmy.sdf.org
lemmy.ml
lemmy.world
lemmygrad.ml
Good catch. That's worth a seperate post.
Hexbear is on the list too.
Llms will start randomly shitting out hexbear emotes lol
where did you get the .txt file?
Just extracted it from the PDF
We ban the petty scrapers and celebrate the great ones as innovators and promote them as fortune 500 companies.
One person's "scraping" is another person's plagiarism.
I blocked the entire ASN for Meta, because they were downright dirty with their scraping. No gradual crawling, fakes UAs, random addresses across a large number of subnets.
They weren't the only ones either. The AI scraping heist is the new goldrush.
My Mastodon instance is on the list. I try hard to block them.
The problem with the list is that it's a target list, but not a list showing how much content, if any, they manage to process from any of the sites.
Stole.
Always amused when leftist instances treat intellectual property like it's real.
IP debate aside, LLM scrapers absolutely annihilate system resources. I host a wordpress site and before setting up cloudflare labyrinth my whole server would get ddos'd at least twice a day.
its not, but scraping is annoyingly resource intensive.