Key Takeaways
- Ownership Is the Lever — The control over what AI ingests from you does not live in a setting somewhere. It lives in owning the domain. Rent the channel and the lever is simply not yours to pull.
- Two Switches, Not One — Training access and search access used to be the same crawler. They aren't anymore. You can block a model from training on your work while staying fully eligible for its search citations, but only if you set both switches on purpose.
- The Index/Noindex of the AI Era — You already decide what search engines may index. The same intentional decision now applies to AI: what do you let models learn from you, and what do you gate? On your own domain that's a line in a file. Everywhere else it's someone else's call.
- A Request, Not a Wall — robots.txt asks well-behaved crawlers to stay out. The badly-behaved ones need an edge rule. Know which of your controls are polite requests and which actually enforce.
I Gave a Decade of My Training Data to a Platform
For about ten years, every run, ride, and swim I did went into the same platform. Heart rate, pace, routes, the whole record of how my body adapted to training. I never once thought of it as mine to lose. It was just where the data went.
Then the terms changed. Not dramatically, not with any villainy, just a quiet update to what the platform could do with the aggregate, and a new tier that put features I relied on behind a wall. I had no leverage, because I had no copy of the relationship. The data was on their turf. My decade of signal was an asset on someone else's balance sheet, and I was renting access to my own history.
That is the exact mistake most people and most companies are now making with their written work, one layer up. They pour their best thinking onto a channel they don't own, and they discover, usually too late, that the most important decision about that work was never theirs to make.
The Lever Lives in the Ownership
Here is the mechanism, because everything else follows from it. The power to decide what an AI model may ingest from you does not live in a checkbox. It lives in owning the domain your work is published on.
On a site you own, you hold a real control surface: a robots.txt file that the major AI crawlers read before they take anything, an llms.txt that shapes how models represent you, and, when you need enforcement rather than a polite request, an edge layer that can actually turn a crawler away. On LinkedIn, Medium, or any social platform, you hold none of that. The platform has already decided, in its own commercial interest, which AI systems get access to everything posted there. You are not a party to the negotiation. You are the inventory.
So "own your channel" is not a branding slogan. It is the difference between holding the lever and watching someone else hold it. Everything below assumes you've made the one decision that makes the rest possible: publishing the canonical version of your work somewhere you control.
The Index/Noindex Decision, One Layer Up
You already make a version of this call without thinking about it. Every site owner decides what search engines may index and what they may not, noindex a staging page, keep a thank-you page out of results, let the real content through. It's routine. Nobody frames it as a philosophical stance; it's just channel hygiene.
The AI era adds a second, parallel decision with far higher stakes: what may models learn from you, and what do you hold back? What you publish openly is, functionally, what your competitors and their models get to absorb. Every article you leave fully open to training crawlers is a small deposit into a system that will happily serve a version of your expertise to the next person who asks, including the people you compete with.
That is not an argument for hoarding. Being ingested by search-and-citation crawlers is often exactly what you want: it's how you get named in AI answers. It's an argument for making the decision on purpose, per crawler, the way you already make the index/noindex decision. The tools to do that finally exist. Most people just haven't looked at them yet.
Training and Search Are Two Different Switches
The single most useful thing to understand in 2026 is that "AI crawler" is not one thing. The providers have split their bots into two jobs, and you can, and should, treat them differently:
- Training crawlers ingest your content to improve model weights.
GPTBot(OpenAI),ClaudeBot(Anthropic),Google-Extended(Google's training token), andCCBot(Common Crawl, which feeds many models) all fall here. - Search-and-citation crawlers fetch your content to answer a live user query and cite you.
OAI-SearchBot,Claude-SearchBot, andPerplexityBotare the ones that make you eligible to be named in an AI answer.
The defensible default for most publishers is now clear: make a deliberate call on the training crawlers based on how differentiated your content is, and keep the search-and-citation crawlers on so you stay visible in the answer engines. Google-Extended is the cleanest illustration of why this works: it opts you out of Gemini training without affecting your Google Search indexing at all. Two switches. Set them separately.
A minimal, honest starting point in robots.txt looks like this, block training, keep citations:
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Stay eligible for AI-answer citations
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /That is the whole lever for the well-behaved crawlers, and it lives in one file on a domain you own. There is no equivalent file for the version of you that lives on someone else's platform.
Know Which Controls Are Requests and Which Are Walls
One caveat that separates people who understand this from people who just copied a config: robots.txt is a request, not a wall. It works because the major, reputable crawlers choose to honor it, and the ones named above do. But a scraper that decides to ignore it faces no technical obstacle from a text file. Reports through 2026 have repeatedly flagged crawlers, Bytespider and certain Meta agents among them, that do not reliably respect the standard.
So match the control to the threat. If your goal is to opt out of the reputable model-training pipelines, robots.txt is genuinely sufficient and the right tool. If your goal is to actually prevent a determined scraper from reading a page, you need enforcement at the edge, a Cloudflare rule, a WAF, bot-management that verifies crawler identity rather than trusting a user-agent string. The mistake is assuming a polite request is a locked door. It isn't, and pretending otherwise is how you get surprised.
llms.txt Is the Opposite Tool
It's easy to lump every new AI-era text file together, but llms.txt points the other way, and the distinction is worth getting right. Where robots.txt governs access, llms.txt governs inclusion. It's a curated markdown file at the root of your site that hands a model a clean map of your most valuable pages, so that when it does represent you, at inference time, answering someone's question, it does so from the content you'd choose rather than whatever it happened to scrape.
Think of it as the inclusion side of the same sovereignty. robots.txt is you deciding what stays out; llms.txt is you deciding what gets seen clearly. Both are levers you only get to pull because you own the domain. It is, again, the same story: ownership is what converts a wish about your content into a control you actually hold.
Reach and Ownership Aren't the Same Question
The obvious objection is distribution. LinkedIn and Medium are where the readers are, and a domain you own starts at zero. True, and it's a false choice. The pattern that keeps both is old and boring and correct: publish the canonical version on a channel you own, then syndicate for reach.
Write it on your domain, where you hold the crawler controls and the canonical URL. Then push a copy to LinkedIn, cross-post to Medium, drop the link in the newsletter. You get the distribution of the rented platforms and the ownership of yours, and the AI-era decision about your IP stays where it belongs, with you. What you should never do is let the rented channel become the only home of your best work, because at that point the platform, not you, is answering the question of who gets to train on your thinking.
This is also where the individual case and the company case converge, and where the deeper governance conversation starts. Owning your outbound channel, what you publish, is one half of digital sovereignty. The other half is governing your inbound AI usage: not letting the whole organization pour its proprietary context into a consumer chatbot with no data agreement, which is its own discipline. I've written that side up separately, the strategic case for AI sovereignty and the practical enterprise AI usage policy that keeps a company from going rogue on ChatGPT.
Own the Asset
The training-data lesson took me years to actually learn, and it generalizes cleanly. The value isn't in the individual run or the individual article. It's in the compounding record of your work, and whether that record sits on your balance sheet or someone else's.
Own the channel. Make the training-versus-citation decision on purpose rather than by default. Enforce the parts that need enforcing, and invite models in where inclusion serves you. None of it is expensive or hard; it's a domain and a couple of text files. The expensive thing is the alternative, handing the single most important decision about your IP to a platform that will always, correctly, decide it in their own interest and not yours.
Working out where the ownership and governance lines should sit for your team's AI strategy? Schedule a consultation.
Frequently Asked Questions
What does digital sovereignty actually mean for someone who publishes?
It means owning the channel your work lives on, so you, not a platform, decide what happens to it. The most consequential version of that decision in 2026 is which AI models are allowed to ingest your content. On a domain you control, that's a lever you hold. On a rented platform, it's a lever someone else holds, and they will pull it in their interest, not yours.
Can I really stop AI from training on my content?
On a site you own, you can tell the major, well-behaved training crawlers to stay out via robots.txt. GPTBot, ClaudeBot, Google-Extended, and CCBot all document and honor it. What you cannot do is stop a crawler that ignores robots.txt; those (some scrapers reportedly including Bytespider and certain Meta agents) need an edge or WAF rule to actually block. And on a platform you don't own, you can't do either: you're bound by whatever access that platform has already granted.
If I block AI crawlers, do I disappear from ChatGPT and Google's AI answers?
Not if you do it deliberately. Providers now split training crawlers from search-and-retrieval crawlers. Block GPTBot (training) but allow OAI-SearchBot, and you stay eligible for ChatGPT's cited answers while opting out of training. Google-Extended is the clean example: it opts you out of Gemini training without affecting Google Search indexing at all. The mistake is treating it as one switch, it's two.
What's the difference between robots.txt and llms.txt?
They point in opposite directions. robots.txt is a boundary, it tells crawlers what they may and may not access. llms.txt is an invitation, a curated markdown map, at the root of your site, that points models at your most valuable pages so they represent you accurately at inference time. One governs access; the other shapes inclusion. Owning your domain is what lets you use either. You get neither on a rented channel.
Isn't publishing on LinkedIn or Medium fine because of the reach?
Reach and ownership are different questions, and you can have both. Publish the canonical version on a domain you own, where you hold the crawler controls, and syndicate to LinkedIn or Medium for distribution. What you should not do is make the rented platform the only home of your best thinking, because then you've handed the single most important AI-era decision about your IP to a company whose incentives are not yours.
Does any of this matter if I'm not a big company?
It matters more, not less, for individuals and small teams, because your differentiated knowledge is a larger share of your total value. A large enterprise leaking one workflow into a model's weights loses a sliver. An independent operator whose entire body of work is freely trainable has given away the thing that made them worth hiring. The barrier to owning the channel, a domain and a text file, is trivially low. The cost of not owning it compounds quietly.
Need Expert Technology Guidance?
20+ years leading technology transformations. Get a technology executive's perspective on your biggest challenges.
