Children turn your back lights down Take it in and get it now You get trapped in the rhythm Time it flies Trapped in the rhytm You'll work until you die Business people drink my wine Take my earth and sip it dry But I'm back to the rhythm Work 'til I Back to the rhythm I'll work until I die

Corporate money in this town Fills my plate 'til I fucking frown Well I'm back to the rhythm Work 'til I Back to the rhythm I'll work until I die

And if the businessman Keeps bringing me down I'll pop over and drop that out tonight

It's the song, song of the sell Take a littlе piece of pie or thеy'll bust your mouth You make the rich more rich No it won't be you Pennies for your time and crumbs to chew Song, song of the sell Take a little piece of pie or they'll bust your mouth You make the rich more rich No it won't be you Pennies for your time and crumbs to chew

Oh bless this food to our bodies And our bodies to your service In the company's holy name Amen

Oh bless this food to our bodies And our bodies to your service In the company's holy name Amen (x7)

Oh it's the song Oh it's the song Oh it's the song Oh it's the song, song of the sell Song, song of the sell Song, song of the sell Oh bless this food to our bodies And our bodies to your service (song, song of the sell)

The conservative think tank The Heritage Foundation wrote that the initial 2022 iteration of KOSA did not go far enough, as the bill did not explicitly list transgender health care as a harm.[27][28] The inclusion of the phrase "consistent with evidence-informed medical information"[29] could be used by attorneys general to cherry-pick anti-trans sources as justification since there is no definition of what "evidence-based medical information" can include.[30] Senator Blackburn, co-author of the bill, has argued that some education about racism and the civil rights movement overlaps with critical race theory, which she labels a "dangerous ideology" that can inflict "mental and emotional damage" upon children.[31] She has also explicitly stated that the bill will be used to censor content involving the transgender community.[32] EFF columnist Jason Kelly states that in the framework provided by the bill, that KOSA could be used to censor education about racism in schools since it could be claimed that it impacts mental health.[33]

In September 2023, a video from the Family Policy Alliance showed Blackburn saying that there should be a priority to "protecting minor children from the transgender [sic] in this culture", alongside her promotion for KOSA, stating "This would put a duty of care and responsibility on the social media platforms, and this is where children are being indoctrinated."[34] This drew criticism from LGBT advocacy groups, fearing that the bill would allow LGBT information for minors to be censored. A spokesperson for Blackburn stated that KOSA was not intended to censor LGBT information.[34] To address these concerns, the bill's language was altered so that the "duty of care" only focused on the product design features that influenced minors' behavior with the platforms, and not the content. As a result, several LGBTQ groups, including GLAAD and GLSEN, dropped their opposition to the bill.[35] However, the EFF, Fight for the Future, and the American Civil Liberties Union found the revisions far from adequate, arguing that LGBTQ content could still be suppressed by targeting any design feature that makes that content available.[26][36][37][38]

If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn’t rely on Google’s indexing and search Reddit by using “site:reddit.com,” you will not see any results from the last week. DuckDuckGo is currently turning up seven links when searching Reddit, but provides no data on where the links go or why, instead only saying that “We would like to show you a description here but the site won't allow us.” Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward. Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google.

The news shows how Google’s near monopoly on search is now actively hindering other companies’ ability to compete at a time when Google is facing increasing criticism over the quality of its search results. And while neither Reddit or Google responded to a request for comment, it appears that the exclusion of other search engines is the result of a multi-million dollar deal that gives Google the right to scrape Reddit for data to train its AI products.

“They’re [Reddit] killing everything for search but Google,” Colin Hayhurst, CEO of the search engine Mojeek told me on a call. Hayhurst tried contacting Reddit via email when Mojeek noticed it was blocked from crawling the site in early June, but said he has not heard back.

“It's never happened to us before,” he said. “Because this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we've never had no reply from anybody before.”

As Jason wrote yesterday, there’s been a huge increase in the number of websites that are trying to block bots that AI companies use to scrape them for training data by updating their robots.txt file. Robots.txt is a text file which instructs bots whether they are or are not allowed to access a website. Googlebot, for example, is the crawler or “spider” that Google uses to index the web for search results. Websites with a robots.txt file can make an exception to give Googlebot access, and not other bots, so they can appear in search results that can generate a lot of traffic. Recently Google also introduced Google-Extended, a bot which crawls the web specifically to improve its Gemini apps, so websites can allow Googlebot to crawl but block the crawler Google uses to power its generative AI products.

Robots.txt files are just instructions, which crawlers can and have ignored, but according to Hayhurst Reddit is also actively blocking its crawler.

Reddit has been upset about AI companies scraping the site to train large language models, and has taken public and aggressive steps to stop them from continuing to do so. Last year, Reddit broke a lot of third party apps beloved by the Reddit community when it started charging to access its API, making many of those third party apps too expensive to operate. Earlier this year, Reddit announced that it signed a $60 million with Google, allowing it to license Reddit content to train its AI products.

Reddit’s robots.txt used to include a bunch of jokes, like forbidding the robot Bender from Futurama from scraping it (User-Agent: bender, Disallow: /my_shiny_metal_ass) and specific pages that search engines are and are not allowed to access. “/r*.rss/” was allowed, while “/login” was not allowed.

Today, Reddit’s robots.txt is much simpler and more strict. In addition to a few links to Reddit’s new “public content policies,” the file simply includes the following instruction:

User-agent: *
Disallow: /

Which basically means: no user-agent (bot) should scrape any part of the site. “Reddit believes in an open internet, but not the misuse of public content,” the updated robots.txt file says.

“Unfortunately, we’ve seen an uptick in obviously commercial entities who scrape Reddit and argue that they are not bound by our terms or policies,” Reddit said in June. “Worse, they hide behind robots.txt and say that they can use Reddit content for any use case they want. While we will continue to do what we can to find and proactively block these bad actors, we need to do more to protect Redditors’ contributions. In the next few weeks, we’ll be updating our robots.txt instructions to be as clear as possible: if you are using an automated agent to access Reddit, you need to abide by our terms and policies, and you need to talk to us.”

Reddit appears to have updated its robots.txt file around June 25, after Mojeek’s Hayhurst noticed its crawler was getting blocked. That announcement said that “good faith actors – like researchers and organizations such as the Internet Archive – will continue to have access to Reddit content for non-commercial use,” and that “We are selective about who we work with and trust with large-scale access to Reddit content.” It also links to a guide on accessing Reddit data which plainly states Reddit considers “Search or website ads” as a “commercial purpose” and that no one can use Reddit data without permission or paying a fee.

Google did not respond to a request for comment, but its announcement of the company’s deal with Reddit points out not only how valuable Reddit is for training AI, but what many of us already know: As Google Search gets increasingly worse in turning up relevant search results, one of the best ways to still get them is to add “Reddit” to your search queries, directing Google to a site where real humans have been writing advice and recommendations for almost two decades. There are a lot of ways to illustrate how useful Reddit can be, but I’m not going to do better than this video:

https://www.youtube.com/watch?v=tcJcw55zIcc

The fact that Google is the only search engine that leads users to that information now, and that it is apparently the result of a $60 million deal around AI training data, is another example of the unintended consequences of the indiscriminate scraping of the entire internet in order to power generative AI tools.

“We've always crawled respectfully and we've done it for 20 years. We're verified on Cloudflare, we don't train AI, we're like genuine, traditional genuine searching, we don't do ‘answer engine’ stuff,” Hayhurst said. “Answer engine” is Perplexity’s name for its AI-powered search engine. “The whole point about Mojeek, our proposition is that we don't do any tracking. But people also use us because we provide a completely different set of results.”

Reddit’s deal with Google, Hayhurst said, makes it harder to offer these alternative ways of searching the web.

“It's part of a wider trend, isn't it?” he said. “It concerns us greatly. The web has been gradually killed and eroded. I don't want to make too much of a generalization, but this didn't help the small guys.”