A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers (David Pierce/The Verge)

David Pierce / The Verge:

A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers  —  For decades, robots.txt governed the behavior of web crawlers.  But as unscrupulous AI companies seek out more and more data …