OpenAI will honor a shared robots.txt signal for AI training preferences

OpenAI said it will respect a shared robots.txt signal that lets site operators state AI training preferences once for all participating providers. Today publishers must block or allow each AI crawler by name, a list that grows as new labs launch bots.

The mechanism builds on robots.txt, the decades-old file sites use to guide crawlers. A shared directive means a single line can express a training preference that any participating provider reads, separating training use from other crawling such as search indexing. That distinction matters to publishers who want visibility in AI search but not in training data.

The move is a response to years of pressure from news organizations and creators, plus lawsuits over training data. Committing to a common standard lets OpenAI appear cooperative while pushing the industry toward one opt-out mechanism instead of many bilateral licensing fights. Its value depends on how many other providers participate; a standard honored by only one company does little.

Caveats: robots.txt is voluntary and only affects future crawling, not data already collected. The announcement does not settle what counts as training versus retrieval for answers. Publishers will want to see audits or enforcement, and whether Google, Anthropic and Meta sign on.