Nieman Journalism Lab

News publishers limit Internet Archive access due to AI scraping concerns

Outlets like The Guardian and The New York Times are scrutinizing digital archives as potential backdoors for AI crawlers.
Nasdaq

Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model

SAN FRANCISCO--(BUSINESS WIRE)-- Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers ...
AOL

A new web crawler launched by Meta last month is quietly scraping the internet for AI training data

Meta has quietly unleashed a new web crawler to scour the internet and collect data en masse to feed its AI model. The crawler, named the Meta External Agent, was launched last month according to ...
Marketplace

Website bots could help publishers fight off traffic loss from AI crawling

Internet infrastructure company Cloudflare said this week it’s launching a system to block bots from scraping clients’ sites or at least allow them to charge AI companies for access. These AI bots ...
Seeking Alpha

Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model

Empowers leading publishers and AI companies to stop the scraping and use of original content without permission SAN FRANCISCO--(BUSINESS WIRE)-- Cloudflare, Inc. (NET) (NYSE: NET), the leading ...