LINKDING

Login

With the rise of AI, web crawlers are suddenly controversial - The Verge

https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders Internet Archive

Tags: #AI #robotstxt #web
Date added: Feb. 15, 2024, 9:53 p.m.
Description: For decades, a humble text file governed the behavior of web scrapers. But as the AI industry grows, the social contract of robots.txt is falling apart.