LINKDING
Shared bookmarks
Shared bookmarks
Sort by
Added ↑
Added ↓
Title ↑
Title ↓
Apply
Tags
With the rise of AI, web crawlers are suddenly controversial - The Verge
#web
#AI
#robotstxt
|
For decades, a humble text file governed the behavior of web scrapers. But as the AI industry grows, the social contract of robots.txt is falling apart.
6 months ago
|
View
Shared by
bacardi55
Previous
1
Next
User
Everyone
bacardi55
Apply
Tags
-robotstxt
A
I
w
eb