linkding

With the rise of AI, web crawlers are suddenly controversial - The Verge

#AI #robotstxt #web | For decades, a humble text file governed the behavior of web scrapers. But as the AI industry grows, the social contract of robots.txt is falling apart.

8 months ago | View Shared by bacardi55

LINKDING

Shared bookmarks

User

Tags