this post was submitted on 22 Aug 2025
151 points (98.7% liked)

Technology

74289 readers
1 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
151
New Threat Research Reveals AI crawlers make up almost 80% of AI bot traffic, Meta Leads AI Crawling As ChatGPT Dominates Real-Time Web Traffic (www.fastly.com)
submitted 8 months ago by Pro@programming.dev to c/technology@lemmy.world
8 comments fedilink hide all child comments
 

cross-posted from: https://programming.dev/post/36083939

top 8 comments
sorted by: hot top controversial new old
[–] PattyMcB@lemmy.world 19 points 8 months ago

I couldn't click the cookie policy garbage because the AI chat bot obscured the buttons on that page.

Talk about AI slop

[–] friend_of_satan@lemmy.world 9 points 8 months ago (2 children)

Soooo what's the other 20% if not crawlers?

[–] INHALE_VEGETABLES@aussie.zone 5 points 8 months ago

It's me replying

[–] Grostleton@lemmy.dbzer0.com 2 points 8 months ago* (last edited 8 months ago)

3 Indian tech workers in a trench coat.

[–] dan@upvote.au 1 points 8 months ago* (last edited 8 months ago) (2 children)

Are they sure that the Meta traffic is AI bots? A lot of people share links in posts on Meta products, which scrape the links to generate a preview.

[–] morto@piefed.social 9 points 8 months ago (1 children)

Aren't those requests made from client devices?

[–] dan@upvote.au 6 points 8 months ago* (last edited 8 months ago)

They are for anything that's E2E encrypted such as Messenger and WhatsApp, so that the link isn't revealed to Meta servers.

As far as I know, for things like posts on Facebook, the server does the scraping. This is especially the case in the web version, since client-side scraping of arbitrary sites would require those sites to have an open CORS policy, which comes with security risks.

There's a Sharing Debugger tool on the Facebook developer site that lets you force rescrape a site.

[–] bert_macklin_fbi@lemmy.world 2 points 8 months ago

It would not surprise me to see that the traffic traced back to "scrapers". A user's perceived relevance often ties to the recency of the dataset.