OpenAI and Google back a shared `robots.txt`
A common robots.txt policy token aims to let website owners state training preferences across participating AI providers.
Website owners have been managing AI preferences one provider at a time. A shared control could reduce that growing list of separate instructions.
What happened
On 30 October, OpenAI and Google backed a common approach to expressing AI training preferences through `robots.txt`, the file websites use to communicate rules to automated visitors.
The mechanism uses a shared policy token developed through the IETF’s AIPREF work. Google said it plans to interpret a training opt-out as a Google-Extended restriction when the website has not explicitly specified Google-Extended separately.
Why it matters
A common signal could make a publisher’s preference easier to maintain as more AI companies collect web content. It also creates a more consistent instruction for participating providers to interpret.
The practical value depends on provider adoption and how those preferences are implemented. Website owners still need to understand the relationship between training controls and other uses of their content.
The bigger picture
A preference signal is not a technical access lock. It communicates intended use rather than physically preventing collection or creating a payment arrangement. The next question is whether shared controls lead to consistent, understandable behaviour across providers.