X updates its terms to ban crawling and scraping (techcrunch.com)

submitted 1 year ago by slin@feddit.de to c/enoughmuskspam@lemmy.world

1 comments fedilink hide all child comments

cross-posted from: https://kbin.social/m/technology@lemmy.world/t/445789

The new terms, which are effective from September 29, ban any kind of scraping or crawling without “prior written consent.”

NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.

The previous version of the terms allowed crawling in accordance with robots.txt.

“NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without our prior consent is expressly prohibited,” it read.

In the last few months, Twitter has also altered its robots.txt file — a file that gives instructions to robot crawlers about what parts of the site they are permitted to visit — to remove instructions for all crawler bots apart from Google.

In 2015, Twitter confirmed that it had a firehose deal in place with Google to surface tweets in search results. It is not clear if the nature or terms of that deal have changed under the new management.

top 1 comments

sorted by: hot top controversial new old

[-] Mautobu@victoriagaming.ca 3 points 1 year ago

I'll do fucking both out of spite.

this post was submitted on 11 Sep 2023

18 points (100.0% liked)

Enough Musk Spam

2089 readers

812 users here now

For those that have had enough of the Elon Musk worship online.

No flaming, baiting, etc. This community is intended for those opposed to the influx of Elon Musk-related advertising online. Coming here to defend Musk or his companies will not get you banned, but it likely will result in downvotes. Please use the reporting feature if you see a rule violation.

Opinions from all sides of the political spectrum are welcome here. However, we kindly ask that off-topic political discussion be kept to a minimum, so as to focus on the goal of this sub. This community is minimally moderated, so discussion and the power of upvotes/downvotes are allowed, provided lemmy.world rules are not broken.

Post links to instances of obvious Elon Musk fanboy brigading in default subreddits, lemmy/kbin communities/instances, astroturfing from Tesla/SpaceX/etc., or any articles critical of Musk, his ideas, unrealistic promises and timelines, or the working conditions at his companies.

Tesla-specific discussion can be posted here as well as our sister community /c/RealTesla.

founded 1 year ago

MODERATORS

jordanpeterson@lemmy.world

an0nym0us@lemmy.dbzer0.com