beehiiv AI Crawl Control: Which AI Bots to Block and Which to Keep (2026)
beehiiv Pro can block 22 AI and search crawlers at the Cloudflare level. Here's what each category does, what blocking costs you, and a sensible default.
beehiiv’s AI Crawl Control is a dashboard in your website settings. It shows which AI bots visit your newsletter website and lets you block them one by one. Blocks are enforced by Cloudflare at the network level, so a blocked bot never gets your pages, whether or not it obeys robots.txt.
The hard part isn’t the setting itself. It’s deciding what to block. Some of the 22 bots on beehiiv’s list only collect training data. Others decide whether your posts get cited in ChatGPT search, Perplexity, or Claude. Four are ordinary search engines, and blocking them hurts your Google or Bing traffic. This guide goes through each group, says what blocking costs you, and suggests a default for most newsletters.
Our sources: beehiiv’s help articles on AI Crawl Control, SEO settings, and LLMs.txt (all updated Oct 1, 2026), beehiiv’s pricing page and a Dec 17, 2025 product update, plus the crawler documentation published by Google, OpenAI, Anthropic, Perplexity, and Apple. We checked all of them on October 27, 2026.
How we researched this: Everything below comes from beehiiv’s help center, pricing page, and product updates, plus each AI company’s own crawler docs. We haven’t used AI Crawl Control in a beehiiv account and haven’t measured traffic before and after blocking. So there are no screenshots or test results here. Where the docs disagree or leave gaps, we say so.
About our links: beehiiv links on this page are affiliate links (/go/beehiiv). That’s the only program we earn from here. Our recommendations would be the same either way.
The short version
| Question | Answer (per today’s sources) |
|---|---|
| What is it? | A dashboard that shows AI crawler traffic to your beehiiv website and lets you block individual crawlers |
| Which plans? | Pro and Enterprise (pricing page row: “AI bot control”). Not on Free or Lite |
| Other requirements | beehiiv’s Website Builder and a custom web domain. Sites on yourpublication.beehiiv.com aren’t eligible |
| Where? | Website → Builder → Settings icon → Site settings → AI Crawl Control |
| How is it enforced? | By Cloudflare, before the request reaches your site, even for bots that ignore robots.txt |
| How many bots? | 22, in four groups: AI training (10), AI search (3), AI assistants (5), search engines (4) |
| Default | Everything allowed. Turning a crawler’s toggle on blocks it |
| SEO impact | Blocking training bots: none, per beehiiv. Blocking search engines: you’ll drop out of their results over time |
| Removes data already collected? | No. It only stops future crawling |
| Affects email? | No. Only your custom web domain |
Our suggested default: block the AI training bots, keep AI search, AI assistants, and search engines allowed. Details below.
Requirements before you look for the setting
beehiiv’s help article lists three conditions. You need all of them:
- A Pro or Enterprise plan. The pricing page’s Website section shows AI bot control (and Custom robots.txt) as ”—” on Free and Lite and “Yes” on Pro and Enterprise.
- The Website Builder. The article says it’s for publications “using the Website Builder.”
- A custom web domain. beehiiv puts the Cloudflare rules on your custom domain, so the default beehiiv subdomain doesn’t qualify. Custom domains start on Lite. Our beehiiv custom domain guide covers the current Settings → Domains setup.
The setting only covers your web domain. beehiiv says it doesn’t touch your email sending domain, branded link domain, or any other domain in your account.
How to block a crawler
From beehiiv’s article:
- Go to Website → Builder, click the Settings icon, open Site settings, and pick AI Crawl Control.
- The dashboard opens with crawler activity at the top and the crawlers list below. Each row shows the crawler’s name, operator, category, and request count.
- Turn a crawler’s toggle on to block it. This is backwards from most settings: on means blocked. Everything is allowed by default.
- Confirm with Block crawler. For search engines, the dialog warns you that blocking will hurt your visibility in search results.
- Settings save automatically. beehiiv says changes go live “as Cloudflare processes the update, typically within a short period after saving.”
You can change your choices at any time.
What the dashboard shows
The dashboard covers the last 4 weeks by default. A filter bar lets you choose a single crawler or a different time range. beehiiv says it updates “in nearly real time” once bots start visiting.
- Crawler requests: all requests from the bots on beehiiv’s list, blocked or not
- Blocked requests: requests stopped by your settings (0 until you block something)
- AI crawler share: the share of all your site traffic that came from AI crawlers
- Unique crawlers: how many of the 22 bots visited
- Crawler activity chart: served vs blocked requests per day. Use it to confirm a block is working.
- Top crawled pages: which posts bots request most
- Response status codes: 200, 301, 404, 429, and others. Lots of 404s can mean bots are hitting old URLs. Our site redirects guide explains how to fix those.
Tip: check the dashboard for a few weeks before blocking anything. If a bot hardly visits, blocking it won’t change much.
The four crawler groups, and what blocking each one costs
1. AI training crawlers (10): usually safe to block
| Crawler | Operator | What it does (per beehiiv’s catalog) |
|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI’s models |
| ClaudeBot | Anthropic | Collects content to train Claude models |
| CCBot | Common Crawl | Builds the Common Crawl archive, widely used as AI training data |
| Bytespider | ByteDance | Training data for ByteDance’s models |
| Meta-ExternalAgent | Meta | Training data for Meta’s AI models and products |
| Amazonbot | Amazon | Improves Amazon services and trains its AI models |
| Diffbot | Diffbot | Feeds Diffbot’s Knowledge Graph, which is also sold as training data |
| Cohere AI | Cohere | Training data for Cohere’s models |
| AI2Bot | Allen Institute for AI | Training data for open models such as OLMo |
| PanguBot | Huawei | Training data for Huawei’s PanGu models |
What blocking costs: beehiiv says blocking training crawlers has no impact on your SEO. The AI companies agree that these bots are separate from their search bots. OpenAI says a site can allow OAI-SearchBot for search while blocking GPTBot from training, and that “each setting is independent.” Anthropic says blocking ClaudeBot tells it to leave your future content out of training.
What it won’t do: it doesn’t remove anything already collected. beehiiv says you’d have to contact each AI company for that.
Side note on Amazonbot: we looked at the robots.txt of one beehiiv-hosted site on Oct 27, 2026. Its header said “beehiiv default robots.txt,” and it already disallowed Amazonbot for the whole site. A robots.txt rule only asks the bot to stay away; AI Crawl Control actually enforces the block.
2. AI search crawlers (3): keep these if you want AI citations
| Crawler | Operator | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexes sites to appear and be cited in ChatGPT search |
| PerplexityBot | Perplexity | Indexes content to be cited in Perplexity answers |
| Claude-SearchBot | Anthropic | Indexes content to improve search results and citations in Claude |
What blocking costs: probably your citations in that product. OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.” Perplexity says PerplexityBot surfaces and links sites in its results and is “not used to crawl content for AI foundation models.” Anthropic says blocking Claude-SearchBot “may reduce your site’s visibility and accuracy” in its search results.
If your newsletter grows from discovery, AI search citations send readers your way. Blocking these bots mostly hurts you, and since they say they don’t train models on what they collect, blocking them doesn’t really protect your writing either.
3. AI assistant crawlers (5): keep unless you have a specific reason
| Crawler | Operator | What it does |
|---|---|---|
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about or links to it |
| Claude-User | Anthropic | Same, for Claude users |
| Perplexity-User | Perplexity | Same, for Perplexity users |
| DuckAssistBot | DuckDuckGo | Fetches pages for DuckDuckGo’s DuckAssist answers |
| MistralAI-User | Mistral AI | Fetches pages for Le Chat users |
These bots only visit when a person asks for your page. This is where AI Crawl Control does something robots.txt can’t. OpenAI says that for ChatGPT-User, “robots.txt rules may not apply” because a user started the request. Perplexity says Perplexity-User “generally ignores robots.txt rules.” A Cloudflare-level block stops them anyway.
What blocking costs: when a reader pastes your post into ChatGPT or Claude and asks for a summary, the assistant can’t read it. beehiiv says blocking assistant bots won’t affect normal search rankings, but it “will impact your ability to show in AI-assisted searches.” For most newsletters, it’s good when a reader’s assistant can read your public posts. It’s better to block these only if your public pages contain content you really don’t want summarized. Gating that content behind a subscription is the stronger tool for that.
4. Search engine crawlers (4): don’t block
| Crawler | Operator | What blocking does (per beehiiv) |
|---|---|---|
| Googlebot | Removes your site from Google Search | |
| Bingbot | Microsoft | Removes your site from Bing and Copilot |
| Applebot | Apple | Affects Siri, Spotlight suggestions, and Apple’s AI features |
| DuckDuckBot | DuckDuckGo | Reduces visibility in DuckDuckGo results |
beehiiv’s warning is direct: blocking these reduces your search visibility, and “over time, your content will stop appearing in search results.” It also notes that a blocked search engine can’t pick up changes, so your titles and descriptions there go stale.
Why you can’t use this to opt out of Google’s AI Overviews: Google says AI is “integral to how Search functions,” so robots.txt rules for Googlebot are the control for AI features in Search. There’s no separate AI Overviews crawler. Blocking Googlebot to avoid AI Overviews means leaving Google Search entirely.
The gap: Google-Extended and Applebot-Extended aren’t crawlers
Google and Apple both offer a way to say “index me for search, but don’t use my content for AI training.” It isn’t a bot you can block. Both are robots.txt tokens:
- Google-Extended. Google says it “doesn’t have a separate HTTP request user agent string.” Crawling is done by Google’s existing crawlers, and the token only works through robots.txt. Disallowing it controls whether your content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”
- Applebot-Extended. Apple says it “does not crawl webpages.” Disallowing it opts your content out of training Apple’s foundation models, and pages can still appear in Apple’s search results.
A network block can’t target a bot that never makes its own requests. So AI Crawl Control can’t do this. beehiiv’s catalog lists Google’s AI component under the Search engine category, which in practice means Googlebot. To opt out of Gemini or Apple training while staying in search, you need robots.txt.
beehiiv’s pricing page lists Custom robots.txt as a Pro and Enterprise feature, and a Dec 17, 2025 beehiiv product update says custom robots.txt lives in your website’s SEO settings. The default file we saw says you can customize it or upload your own. beehiiv’s help center doesn’t have a step-by-step article on editing it yet, so check the in-app screen. If you add rules, add these groups and leave beehiiv’s existing lines in place:
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /These only ask, they don’t enforce. But Google and Apple both say they follow them, and robots.txt is the only way they offer to send this opt-out.
How it fits with beehiiv’s other AI settings
| Setting | Plans | What it does | Enforced? |
|---|---|---|---|
| Discoverable on the web (SEO & sharing) | All plans | Site-wide or per-page switch. Off updates robots.txt to ask all search engines and AI bots not to index | No. It relies on bots following robots.txt |
| Custom robots.txt | Pro, Enterprise | Your own robots.txt rules, e.g. Google-Extended | No. Same reason |
| AI Crawl Control | Pro, Enterprise + custom domain | Blocks specific crawlers from the 22-bot list | Yes, by Cloudflare |
| LLMs.txt | All paid plans | A summary file that helps AI tools understand your site; auto-generated, editable | N/A. It guides bots, it doesn’t block them |
| AEO settings | Pro, Enterprise | Per-post structured data | N/A. It describes your content |
A few details from beehiiv’s docs:
- Discoverable on the web is all-or-nothing. A page can turn itself off when the site is on, but it can’t turn itself back on when the site is off. beehiiv’s FAQ suggests this toggle if you want to block every crawler, search engines included, in one step. Google says robots.txt alone doesn’t keep a URL out of its index; our guide to hiding a beehiiv post or page from Google covers what does.
- LLMs.txt is generated automatically from up to 50 posts plus pages, authors, categories, products, and podcast episodes. It’s capped at 6,144 characters, and older posts get cut from the end. You can paste or upload your own and Reset to default later. It’s at yourdomain.com/llms.txt. More in our beehiiv LLMs.txt guide.
- AEO settings don’t decide which bots get in. They describe what a post is about. Our beehiiv AEO settings guide explains which schema blocks Google still uses.
Three setups to copy
1. Most newsletters (reach plus some protection)
- Block: all 10 AI training crawlers
- Allow: AI search, AI assistants, search engines
- Optional: custom robots.txt with Google-Extended and Applebot-Extended disallowed
2. Maximum reach (growth over everything)
- Block nothing. Keep Discoverable on the web on and the default LLMs.txt
- Still look at the dashboard now and then for odd traffic spikes
3. Premium or paid-content-heavy publication
- Block: training crawlers, and consider the assistant crawlers if your free posts are previews of paid work
- Allow: search engines, so new readers can still find you
- Keep your best material gated. Bot settings only cover public pages
Whatever you pick, check the Blocked requests card and the activity chart a day or two later to confirm the blocks are working.
What it costs
AI bot control is a Pro and Enterprise feature. Pro prices from beehiiv’s pricing page, checked October 27, 2026:
| Subscribers | Lite (billed yearly / monthly) | Pro (billed yearly / monthly) |
|---|---|---|
| 1,000 | $49 / $59 per month | $95 / $109 per month |
| 2,500 | $69 / $79 per month | $129 / $149 per month |
| 5,000 | $95 / $109 per month | $169 / $199 per month |
| 10,000 | $119 / $139 per month | $199 / $239 per month |
Lite doesn’t include AI bot control or custom robots.txt. beehiiv’s pricing FAQ says Lite and Pro come with a free trial and no credit card, but the AI Crawl Control article doesn’t say whether the feature works during a trial. You’d also need a custom domain connected before you could try it. Full plan breakdown: beehiiv pricing explained.
Where beehiiv’s docs disagree or leave gaps
- Blocking by category. The article’s intro says you can block crawlers “one at a time or by category.” Its FAQ says “You can block each crawler individually,” and points to Discoverable on the web if you want to block everything at once. Expect to click through the training bots one by one.
- “Google’s extended crawler.” beehiiv’s Search engine category says it includes “some crawlers with an AI component (such as Google’s extended crawler).” Google says Google-Extended isn’t a separate crawler. The only Google entry in beehiiv’s catalog is Googlebot, so you can’t block Google-Extended alone here.
- Bots not on the list. New crawlers only show up once beehiiv adds them. beehiiv says platform-level protection against known abusive bots stays on regardless.
- Custom robots.txt how-to. It’s on the pricing page and in a product update, but the help center has no step-by-step article for it.
- Free trial and downgrades. Neither the feature article nor the pricing page says what happens to your blocks if you drop from Pro to Lite.
Checklist
- Confirm Pro or Enterprise, the Website Builder, and a custom web domain
- Open Site settings → AI Crawl Control and watch traffic for a couple of weeks
- Block the AI training crawlers you don’t want (toggle on = blocked)
- Leave search engines allowed. Think twice before blocking AI search or assistant bots
- Optional: add Google-Extended and Applebot-Extended disallow rules in custom robots.txt
- Check Blocked requests and the chart to confirm the blocks work
- Remember: no effect on data already collected, on email, or on bots beehiiv hasn’t listed
Our take
AI Crawl Control is one of the more useful Pro features for writers who care about how their work is used. It enforces blocks instead of relying on robots.txt, it shows you which bots actually visit, and it can stop user-triggered fetchers that ignore robots.txt. Most newsletters should do the simple thing: block the training bots and leave everything else allowed. That protects your writing without costing search traffic or AI citations. It isn’t a reason to upgrade on its own. If you’re already on Pro with a custom domain, it’s worth setting up.
Try beehiiv · beehiiv AEO settings · beehiiv website setup · beehiiv custom domain · beehiiv pricing explained
Sources
All sources below were checked October 27, 2026.
- beehiiv: How to control which AI crawlers access your content (updated Oct 1, 2026; Pro and Enterprise): requirements (Website Builder, custom web domain), Cloudflare network-level enforcement, settings path, toggle on = block, Block crawler confirmation, dashboard metrics, 22-crawler catalog and categories, FAQ (SEO impact, existing data, email, subdomains, new crawlers)
- beehiiv: SEO settings for your website (updated Oct 1, 2026; All Plans): Discoverable on the web, per-page discoverability override, AI Crawl Control for Pro and Enterprise with a custom domain
- beehiiv: How to use and customize a LLMs.txt file (updated Oct 1, 2026; All Paid Plans): auto-generated file, up to 50 posts, 6,144-character limit, customize and reset
- beehiiv: How to use a custom domain for your publication (updated Oct 23, 2026; All Paid Plans)
- beehiiv: Pricing: Lite and Pro prices at 1,000–10,000 subscribers; Custom robots.txt and AI bot control rows (Pro and Enterprise); free trial FAQ
- beehiiv product update: Automations v3 (Dec 17, 2025): custom robots.txt added to the website’s SEO settings
- Google: Google’s common crawlers: Google-Extended is a robots.txt token with no separate user agent; Gemini training and grounding; no effect on Search inclusion or ranking
- Google Search Central: AI features and your website: robots.txt for Googlebot is the control for AI features in Search
- OpenAI: Overview of OpenAI crawlers: OAI-SearchBot, GPTBot, ChatGPT-User; independent settings; robots.txt may not apply to user-initiated ChatGPT-User visits
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?: ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity: Perplexity crawlers: PerplexityBot not used for foundation model training; Perplexity-User generally ignores robots.txt
- Apple: About Applebot: Applebot-Extended doesn’t crawl and controls training use only; pages remain in search
- Cloudflare: AI Crawl Control docs: the Cloudflare product beehiiv’s feature is built on
- AudienceBench: beehiiv AEO settings, beehiiv website setup, beehiiv custom domain, beehiiv site redirects, beehiiv pricing explained