AI Training Opt-Out: The New Robots.txt Signal Every CMO Needs to Understand

AI search is changing the relationship between content ownership and digital visibility. The new shared training preference signal gives businesses more control, but CMOs need a broader strategy for protecting intellectual property without sacrificing discoverability.

The October 30 announcement from OpenAI, Google, Microsoft, and Anthropic could change how businesses protect proprietary content while staying visible in AI search. Here's what marketing leaders need to understand, what the new signal actually does, and how to prepare.

For years, businesses have been dealing with an uncomfortable question:

How much of your intellectual property are you willing to give away to remain visible online?

Think about the content your company has spent years creating.

Original research. Industry benchmarks. Customer data. Competitive analysis. Product comparisons. Technical documentation. Proprietary frameworks.

These aren't just blog posts. They're business assets. They represent research budgets, employee expertise, customer insights, and, in many cases, the competitive advantage that separates your company from everyone else.

Now imagine an AI platform using that material to answer a potential customer's question. The customer gets the insight, but never visits your website, sees your brand, or knows where the information came from.

That's the tension marketing leaders have been trying to solve.

Until recently, the options were frustrating. Block AI crawlers and risk limiting your visibility. Allow them in and potentially give AI companies access to valuable content for model training.

On October 30, 2026, something important happened.

OpenAI, Google, and Microsoft announced support for a shared AI training preference signal in robots.txt. Anthropic is also reported to support the initiative. The mechanism uses a policy token called __usage_ai-training.

It's an important step toward giving publishers more control over how their content is used.

But there's a bigger business conversation here.

The future of AI search shouldn't require businesses to give up control of the content that makes them valuable.

And CMOs need to start treating AI content governance as part of their marketing strategy, not just an engineering configuration.

The problem: AI visibility and intellectual property have been pulling in opposite directions

Traditional SEO had a relatively straightforward value exchange.

You published useful content. Search engines crawled it, indexed it, and potentially sent visitors to your website.

You provided information. Search engines provided discoverability.

The economics were never perfect, but the relationship was understandable.

Generative AI changed that relationship.

AI platforms can retrieve information from websites, summarize it, combine it with other sources and deliver an answer directly inside their interfaces.

That creates a new challenge for publishers and businesses.

Your content can contribute to an answer without the user ever needing to visit your website.

And there are two different ways AI systems can use that content.

The first is model training.

Content is collected and used to modify the learned parameters of a generative AI model. Once information influences a model's parameters, it isn't necessarily presented as a separately identifiable source every time the model generates an answer.

The second is live retrieval and grounding.

An AI system accesses external information to help generate a response to a user's current question. Depending on the platform, the answer may include citations and links to the original sources.

These are different activities, with different implications for publishers.

The problem is that businesses haven't always had a simple, consistent way to communicate their preferences across AI providers.

For marketing teams, this creates a difficult balancing act:

• Protect valuable research without unnecessarily limiting discoverability.
• Maintain visibility in AI-generated answers without automatically agreeing to every possible use of your content.
• Give legal and compliance teams more control without turning robots.txt into an endless collection of vendor-specific rules.

This is the context that makes the new proposal interesting.

What happened on October 30, 2026?

The development is a proposed shared mechanism for communicating AI training preferences through the existing robots.txt framework.

The proposed policy token is:

__usage_ai-training

Instead of maintaining a separate training preference for every participating AI provider, website owners can use one shared signal.

The basic example looks like this:

`` User-agent: __usage_ai-training Disallow: / ``

In plain English, this communicates that the website operator does not want the relevant content used for generative AI model training.

You can also apply the preference selectively:

`` User-agent: __usage_ai-training Disallow: /research/ Disallow: /benchmarks/ Disallow: /proprietary-data/ Allow: /press-releases/ ``

This expresses a preference against training on the specified directories while allowing the stated exception.

The distinction matters because not every page on your website has the same business value.

Your homepage, product pages, educational articles, and proprietary research shouldn't necessarily be governed by the same content policy.

There is, however, an important technical detail that should not be overlooked.

The proposal is an individual IETF Internet-Draft, not an officially adopted IETF standard. It is part of the ongoing AI Preferences working group discussion and remains subject to change.

IETF Datatracker+1

Which AI companies are supporting it?

The announcements are encouraging, but the implementation timelines are not identical.

• OpenAI: Announced that it intends to honor the shared signal in the coming months. Existing crawler controls will continue to apply.

OpenAI

• Google: Plans to support the shared signal in 2027. Its existing Google-Extended controls will continue to function.

Google Developers

• Microsoft: Plans to begin honoring the new preferences from February 1, 2027, as its crawlers discover them. It is also introducing separate controls for AI training and AI grounding.

Bing Blogs

• Anthropic: Its standards representative authored the Internet-Draft, and the company is reported to support the shared initiative.

The Verge+1

This is meaningful progress, but it isn't an overnight change in how the entire AI ecosystem operates.

For companies managing significant content portfolios, the sensible approach is to prepare for adoption while continuing to monitor provider-specific documentation.

The distinction CMOs need to understand: AI training is not AI search visibility

This is probably the most important part of the entire development.

The new signal addresses the use of content for AI training. It does not automatically control whether an AI search engine can crawl a page, index it, or use it to ground a generated answer.

Those are separate decisions.

Consider a company that publishes an annual report on enterprise software spending.

It wants AI systems to discover the report when someone asks about enterprise software budgets. It wants its research to be referenced as a credible source.

At the same time, the company may not want that report used to train future generative models.

These preferences aren't contradictory.

A company can express a preference against AI training while maintaining separate crawl and search permissions.

But there is an important qualification: the shared token itself does not guarantee that a provider will retrieve, cite, or recommend the report.

That depends on the provider's implementation, search systems, relevance signals, and the user's query.

Training, crawling, and grounding are three different things

It's useful to stop treating all AI bots as if they perform the same job.

AI training

The use of content to modify the learned parameters of a generative model.

The new shared policy token is designed to communicate preferences about this activity.

Crawling

The process of accessing a website and retrieving its pages.

Traditional search crawlers and AI-related crawlers may have different purposes. The new policy token does not replace crawler-specific access rules.

AI grounding and retrieval

The use of external information to help generate an answer.

This can include retrieving a page at query time, using its information to formulate a response and potentially presenting the original source as a citation.

The distinction is particularly important because the IETF draft explicitly treats content-use preferences and crawler-access rules as independent.

In other words, a training opt-out is not the same as a crawl block.

IETF Datatracker+1

For CMOs, this changes the question from:

Should we allow AI bots onto our website?

To:

Which AI activities do we want to permit, for which content assets, and under what commercial conditions?

That's a much more useful conversation.

Why this matters commercially: your content is an asset, not just an SEO input

I think this is where marketing leadership needs to become more involved.

For years, content has been measured primarily through traffic, rankings, backlinks, engagement and conversions.

But enterprise content has another dimension: the value of the information itself.

A proprietary benchmark can influence purchasing decisions.

An original research report can establish category authority.

A technical comparison can help a prospect understand why one solution is better than another.

A customer study can become a reference point across an entire industry.

These assets create competitive advantage because they contain information, analysis or expertise that competitors cannot easily reproduce.

AI changes the distribution of that value.

When an AI system can answer a question using information from your research, the user may receive the benefit without interacting with the company that produced it.

That doesn't automatically mean AI visibility is bad. In fact, being referenced by an AI answer engine can create significant brand value.

But it does mean we need a more sophisticated approach to measuring the return on content investment.

The opportunity is not to block AI. It's to manage the exchange.

I wouldn't recommend that every company simply block AI training across its entire website.

That would be an equally simplistic response to a much more complicated problem.

Some content is designed to be distributed as widely as possible.

For example:

• Educational articles that introduce your category.
• Product explanations that help buyers understand your solution.
• Public documentation that reduces friction during evaluation.
• Thought leadership that builds brand authority.

Other content has a much stronger proprietary component.

For example:

• Original industry benchmarks.
• Research based on proprietary customer datasets.
• Detailed competitive intelligence.
• Paid research reports.
• Unique methodologies and analytical frameworks.

The right policy depends on the business value of each asset, the legal rights attached to it, and the company's commercial objectives.

A good CMO should be asking:

Which content should help AI systems understand our category, and which content represents intellectual property that we need to protect?

That is a content strategy decision, not simply a technical SEO decision.

The new enterprise playbook: what CMOs should do now

The announcement creates an opportunity to review how your company manages content across the AI ecosystem.

I would approach it in five steps.

Step 1: Audit your content based on business value

Start by looking at your website as a portfolio of business assets.

Don't treat every URL equally.

Identify three broad groups.

Public discovery content

This includes educational articles, product pages, category explanations, and other material designed to increase brand awareness and attract potential customers.

The priority here is discoverability.

Strategic intellectual property

This includes original research, proprietary benchmarks, unique methodologies, and other content that creates a competitive advantage.

The priority here is to define acceptable uses and protect the value of the underlying work.

Restricted or commercially sensitive content

This includes internal documentation, customer information, confidential datasets, and material that should never be publicly accessible.

The priority here is access control, authentication, and security.

One important distinction: robots.txt is not a security mechanism. If content is confidential, it should not be publicly exposed and protected only through crawler instructions.

Step 2: Build a clear AI usage policy

Once you've categorized your content, work with engineering and legal teams to define what the company wants to allow.

For example:

• Public educational content: available for search discovery and evaluated individually for training permissions.
• Proprietary research: potentially available for live retrieval, with training restricted where supported.
• Confidential material: protected through authentication and access controls.
• Licensed datasets: governed by explicit contractual terms.

This gives everyone a common framework.

It also avoids the situation where marketing wants maximum AI visibility, legal wants to block everything, and engineering is left trying to reconcile conflicting instructions.

Step 3: Prepare your robots.txt configuration

The new shared token provides a simpler way to communicate training preferences to participating providers as they implement it.

For example:

```

Shared AI training preference

User-agent: __usage_ai-training Disallow: /research/ Disallow: /benchmarks/ Disallow: /proprietary-data/ ```

This is a preference signal, not a technical firewall.

It should also be reviewed alongside existing crawler-specific rules.

For example, if your company wants to remain accessible to search crawlers, review the relevant rules for:

• OAI-SearchBot
• Googlebot
• PerplexityBot
• Bingbot

Don't assume that adding the shared training token automatically overrides an existing crawler restriction.

Also, don't copy a generic robots.txt configuration into production without testing how your current crawler rules interact with it.

The IETF draft specifies that existing crawler rules remain independent of the new content-use preferences.

IETF Datatracker

Step 4: Strengthen protection beyond robots.txt

Here's something I don't want marketing leaders to overlook.

robots.txt communicates preferences to compliant crawlers. It does not prevent an unauthorized scraper from accessing publicly available content.

If your company has valuable research or commercially sensitive material, you need additional protection.

Work with your security and infrastructure teams on:

• Web Application Firewall (WAF) policies.
• Bot management and traffic analysis.
• Rate limiting for suspicious request patterns.
• Authentication for restricted research and datasets.
• Monitoring unusual scraping behavior.
• Logging and alerting for repeated access to sensitive directories.

Cloudflare, AWS, and Fastly offer infrastructure-level capabilities that can help with these controls.

But don't confuse bot detection with perfect bot identification. User-agent strings can be spoofed, and IP-based identification has limitations.

The goal is layered protection, not the assumption that one firewall rule will solve the problem.

Step 5: Measure AI visibility and IP protection separately

This is where marketing analytics needs to evolve.

If you're measuring only organic traffic, you're missing a growing part of the discovery journey.

And if you're measuring only AI mentions, you're missing the commercial value of the content itself.

I would establish two separate measurement areas.

AI discovery and commercial visibility

• Brand mentions across relevant AI answer engines.
• Citation frequency for high-value research and product pages.
• Share of visibility across priority category questions.
• Referral sessions from AI platforms.
• Assisted conversions and qualified pipeline influenced by AI discovery.
• Conversion rates from AI-referred visitors.

Content governance and IP protection

• Percentage of high-value assets with documented usage policies.
• Coverage of training preferences across relevant directories.
• Compliance with provider-specific crawler requirements.
• Unauthorized scraping incidents.
• Number of restricted assets exposed publicly.
• Licensing agreements and approved data partnerships.

The objective is not simply to maximize AI citations.

It's to understand whether your content is creating measurable business value while remaining under the level of control your company requires.

The Microsoft announcement adds another layer: training and grounding can have separate permissions

One detail in the October 30 announcements deserves particular attention.

Microsoft is going beyond a shared training preference.

Its proposed publisher controls distinguish between AI training and AI grounding.

That means a publisher can make separate decisions about whether content can be used to train generative models and whether it can be used to inform AI-generated responses.

Microsoft also plans to introduce page-level controls for these preferences.

The distinction is commercially important.

Imagine you publish a proprietary financial benchmark.

You might be comfortable allowing Copilot to reference the benchmark in an answer, provided it links to the original research. But you may not want that content used for future model training.

Or you might have a commercial agreement that allows certain uses but excludes others.

Separate controls create more room for those decisions.

Microsoft says its new preferences will begin taking effect as its crawlers discover them from February 1, 2027. It also warns that restricting grounding can reduce visibility in its AI experiences.

Bing Blogs

This is a direction worth watching because it moves the conversation beyond a simple yes-or-no approach to AI.

We're moving toward a world where publishers can express more granular preferences about how their content is used.

And that has implications for content licensing, enterprise data strategy, and the economics of digital publishing.

What this means for GEO and the future of organic acquisition

There's a tendency to discuss Generative Engine Optimization (GEO) as if it's simply traditional SEO with a new set of keywords.

I think that's a mistake.

AI search introduces a different relationship between content, discovery, and the user journey.

In traditional search, a user might see ten results, click a link, and explore a website.

In generative search, the user may receive a synthesized answer, compare several recommendations, and make a decision without visiting the original websites.

That changes what it means to win visibility.

Being indexed is not the same as being cited.

Being cited is not the same as being recommended.

Being recommended is not the same as generating a qualified lead.

And generating a qualified lead is not the same as closing revenue.

The new training preference signal doesn't solve these challenges. It doesn't guarantee citations, recommendations, or traffic.

But it gives companies another mechanism for managing one important part of the relationship: whether their content should be used for AI training.

For GEO teams, I would focus on four areas.

1. Make original expertise easy to identify.

AI systems need to understand what your company knows, what it publishes, and why its information deserves attention.

Clear authorship, transparent sourcing, original data, and well-structured content all help establish credibility.

1. Publish information that is difficult to replace.

Generic content is increasingly easy to reproduce.

Original research, proprietary insights, expert commentary, detailed case studies, and first-party evidence give your brand something more meaningful to contribute.

1. Build a clear connection between information and your brand.

Don't publish valuable research without making its origin obvious.

Use consistent company attribution, named authors, publication dates, methodology notes, and links to supporting evidence.

If your benchmark becomes a reference point in your industry, people should know who created it.

1. Measure visibility beyond the click.

Track where your company appears in relevant AI answers, whether your original research is cited, and whether those interactions influence consideration and pipeline.

The goal isn't to chase every AI mention.

It's to become a trusted source in the conversations that matter to your business.

The CMO and General Counsel checklist

Before changing your company's AI crawling and content policies, I would bring marketing, legal, engineering, and security into the same conversation.

Here is the checklist I'd use.

• Inventory your content assets. Identify proprietary research, original datasets, commercial reports, product documentation, and public marketing content.
• Classify content by business value. Define which assets should be freely discoverable and which require specific usage restrictions.
• Review existing crawler policies. Audit robots.txt and current rules for GPTBot, Google-Extended, ClaudeBot, and other relevant crawlers.
• Prepare for the shared training signal. Evaluate the __usage_ai-training token and monitor provider implementation timelines.
• Protect live discovery where appropriate. Check that existing search and AI retrieval permissions reflect your visibility objectives.
• Review grounding preferences. Pay particular attention to Microsoft's separate training and grounding controls.
• Strengthen infrastructure security. Use authentication, WAF policies, bot management, and rate limiting where appropriate.
• Define an AI licensing position. Establish when third parties can use proprietary research through commercial agreements, APIs, or explicit licensing arrangements.
• Create an AI visibility measurement framework. Track citations, brand mentions, referral traffic, and business outcomes.
• Assign ownership. Make it clear who is responsible for maintaining these policies and reviewing them as the ecosystem changes.

One final point: don't rush to remove existing crawler restrictions just because a shared signal has been announced.

Implementation is still rolling out, and each provider has its own rules and timelines.

Review the official documentation before changing production configurations.

My take: the next competitive advantage is controlling how your expertise travels

I see this development as part of a much larger shift in digital marketing.

For a long time, companies competed to own the first page of Google.

Then they competed for featured snippets, rich results, and increasingly expensive paid placements.

Now they're competing to become the sources AI systems trust when answering questions.

But there's a difference between making your expertise discoverable and surrendering control over how that expertise is used.

Those two things should never have been treated as the same decision.

The October 30 proposal is not a complete solution. It's not an adopted global standard, it doesn't prevent unauthorized scraping, and it doesn't guarantee that an AI engine will cite your content.

But it represents meaningful progress toward a more structured relationship between publishers and AI providers.

For CMOs, the bigger lesson is that content strategy needs to evolve.

Your website is no longer just a destination for organic traffic. It's also a repository of proprietary knowledge, a source of information for AI systems, and an increasingly important part of your company's digital intellectual property.

That means content governance belongs in the boardroom alongside brand strategy, demand generation and data ownership.

My recommendation: don't build an AI strategy around blocking everything or allowing everything. Build one around knowing what you own, what you're willing to share, what you want to protect, and what commercial value you expect in return.

Because in the AI search economy, the question isn't just whether people can find your expertise.

It's whether your company still owns the value that expertise creates.

FAQ

What is the __usage_ai-training robots.txt token?

It's a shared policy token described in an October 30, 2026 IETF Internet-Draft. Website operators can use it in a robots.txt group to communicate whether their content may be used for generative AI model training. Participating providers have announced plans to honor the signal on different implementation schedules.

Does the new AI training opt-out block ChatGPT Search or Google AI Overviews?

No. The training preference is separate from crawler access and AI search grounding. However, existing crawler restrictions and provider-specific policies can affect whether content is discoverable or used in AI-generated answers. The shared token does not guarantee search visibility or citations.

Is __usage_ai-training an official IETF standard?

No. As of October 31, 2026, it is an individual IETF Internet-Draft submitted to the AI Preferences working group. It is not an adopted RFC or an approved IETF standard.

How do I block AI training using robots.txt?

The proposed syntax is: </> http User-agent: __usage_ai-training Disallow: / You can also apply restrictions to individual directories, using Allow exceptions where appropriate. Check each provider's implementation guidance before relying on the signal.

Can I allow AI search while blocking AI model training?

Yes, the two preferences are designed to be independent. You can express a preference against training while maintaining separate crawler permissions for search discovery. Actual retrieval and citation behavior depends on each provider.

Does robots.txt prevent AI companies from scraping my website?

No. robots.txt communicates preferences and crawler rules to compliant operators. It is not an access-control or security mechanism. Protect restricted information with authentication, authorization, infrastructure security, and appropriate contractual controls.

What is the difference between AI training and AI grounding?

AI training uses content to modify the learned parameters of a generative model. AI grounding uses external content to inform an answer being generated. Grounding may result in a citation or link to the original source, depending on the platform.

Why should CMOs care about AI content governance?

Because company content increasingly has two forms of value: its ability to attract and convert customers, and the proprietary information and expertise it contains. AI content governance helps marketing leaders balance discoverability, intellectual property protection, brand visibility and commercial licensing opportunities.

When will the shared AI training signal become available?

Implementation varies by provider. OpenAI has announced plans for the coming months, Google is targeting 2027, and Microsoft has announced February 1, 2027 as the start of honoring the new preferences as its crawlers discover them. Provider documentation should be checked for updates.

The future of AI search isn't about choosing between visibility and ownership. It's about making your expertise discoverable without giving away control of the value that makes your business different. Let's Talk.